taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

taler-developer-manual.rst (65653B)


      1 ..
      2   This file is part of GNU TALER.
      3 
      4   Copyright (C) 2014-2025 Taler Systems SA
      5 
      6   TALER is free software; you can redistribute it and/or modify it under the
      7   terms of the GNU Affero General Public License as published by the Free Software
      8   Foundation; either version 3.0, or (at your option) any later version.
      9 
     10   TALER is distributed in the hope that it will be useful, but WITHOUT ANY
     11   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
     12   A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
     13 
     14   You should have received a copy of the GNU Affero General Public License along with
     15   TALER; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
     16 
     17   @author Christian Grothoff
     18 
     19 General Developer Manual
     20 ########################
     21 
     22 .. note::
     23 
     24   This manual contains information for developers working on GNU Taler
     25   and related components.  It is not intended for a general audience.
     26 
     27 
     28 Project Overview
     29 ================
     30 
     31 GNU Taler consists of a large (and growing) number of components
     32 in various Git repositories.  The following list gives a first
     33 overview:
     34 
     35 * exchange: core payment processing logic with a REST API, plus various
     36   helper processes for interaction with banks and cryptographic
     37   computations. Also includes the logic for the auditor and an
     38   in-memory "bank" API implementation for testing.
     39 
     40 * libeufin: implementation of the "bank" API using the EBICS protocol
     41   used by banks in the EU.  Allows an exchange to interact with
     42   European banks.
     43 
     44 * taler-magnet-bank: implementation of the "bank" API using the Magnet Bank
     45   API. Allows an exchange to interact with Magnet Bank.
     46 
     47 * taler-cyclos: implementation of the "bank" API using the Cyclos API. Allows an exchange to interact with a Cyclos network.
     48 
     49 * taler-wise: implementation of the "bank" API using the Wise API. Allows an exchange to interact with Wise.
     50 
     51 * depolymerization: implementation of the "bank" API on top of
     52   blockchains, specifically Bitcoin and Ethereum. Allows an exchange
     53   to interact with crypto-currencies.
     54 
     55 * merchant: payment processing backend to be run by merchants,
     56   offering a REST API.
     57 
     58 * wallet-core: platform-independent implementation of a wallet to be run by
     59   normal users. Includes also the WebExtension for various browsers.
     60   Furthermore, includes various single-page apps used by other
     61   components (especially as libeufin and merchant).  Also includes
     62   command-line wallet and tools for testing.
     63 
     64 * taler-android: Android Apps including the Android wallet, the
     65   Android point-of-sale App and the Android casher app.
     66 
     67 * taler-ios: iOS wallet App.
     68 
     69 * sync: backup service, provides a simple REST API to allow users to
     70   make encrypted backups of their wallet state.
     71 
     72 * anastasis: key escrow service, provides a simple REST API to allow
     73   users to distribute encryption keys across multiple providers and
     74   define authorization policies for key recovery.
     75 
     76 * taler-mdb: integration of Taler with the multi-drop-bus (MDB) API
     77   used by vending machines. Allows Taler payments to be integrated
     78   with vending machines.
     79 
     80 * gnu-taler-payment-for-woocommerce: payment plugin for the
     81   woocommerce (wordpress) E-commerce solution.
     82 
     83 * twister: man-in-the-middle proxy for tests that require fuzzing a
     84   REST/JSON protocol.  Used for some of our testing.
     85 
     86 * challenger: implementation of an OAuth 2.0 provider that can be used
     87   to verify that a user can receive SMS or E-mail at particular addresses.
     88   Used as part of KYC processes of the exchange.
     89 
     90 * taler-mailbox: messaging service used to store and forward payment
     91   messages to Taler wallets.
     92 
     93 * taldir: directory service used to lookup Taler wallet addresses for
     94   sending invoices or payments to other wallets.
     95 
     96 * taler-merchant-demos: various demonstration services operated at
     97   'demo.taler.net', including a simple shop and a donation page.
     98 
     99 There are other important repositories without code, including:
    100 
    101 * gana: Hosted on git.gnunet.org, this repository defines various
    102   constants used in the GNU Taler project.
    103 
    104 * docs: documentation, including this very document.
    105 
    106 * marketing: various presentations, papers and other resources for
    107   outreach.
    108 
    109 * large-media: very large data objects, such as videos.
    110 
    111 * www: the taler.net website.
    112 
    113 Fundamentals
    114 ============
    115 
    116 Versioning
    117 ----------
    118 
    119 A central rule is to never break anything for any dependency. To accomplish
    120 this, we use versioning, of the APIs, database schema and the protocol.  The
    121 database versioning approach is described in the :ref:`Database schema
    122 versioning <DatabaseVersioning>` section.  Here, we will focus on API and
    123 protocol versioning.
    124 
    125 The key issue we need to solve with protocols and APIs (and that does not
    126 apply to database versioning) is being able to introduce and remove features
    127 without requiring a flag day where all components must update at the same
    128 time. For this, we use GNU libtool style versioning with MAJOR:REVISION:AGE
    129 and *not* semantic versioning (SEMVER).  With GNU libtool style versioning,
    130 first the REVISION should be increased on every change to the respective code.
    131 Then, each time a feature is introduced or deprecated, the MAJOR and AGE
    132 numbers are increased. Whenever an API is actually removed the AGE number is
    133 reduced to match the distance since the removed API was deprecated.  Thus, if
    134 some client implements version X of the protocol (including not using any APIs
    135 that have been deprecated), it is compatible for any implementation where
    136 MAJOR is larger or equal to X, and MAJOR minus AGE is smaller or equal to X.
    137 REVISION is not used for expected compatibility issues and merely serves to
    138 uniquely identify each version (in combination with MAJOR).
    139 
    140 To evolve any implementation, it is thus critical to first of all never
    141 just break an existing API or endpoint. The only acceptable modifications
    142 are to return additional information (being aware of binary compatibility!)
    143 or to accept additional optional arguments (again, in a way that does not
    144 break existing users). Thus, the most common way to introduce changes will
    145 be the addition of new endpoints. Breaking existing endpoints is only ever
    146 at best acceptable while in the process of introducing it and if you are
    147 absolutely sure that there are zero users in other components.
    148 
    149 When removing endpoints (or fields being returned), you must first deprecate
    150 the existing API (incrementing MAJOR and AGE) and then wait for all clients,
    151 including all clients in operation (e.g. Android and iOS Apps, e-commerce
    152 integrations, etc.) to upgrade to a protocol implementation above the
    153 deprecated MAJOR revision. Only then you should remove the endpoint and reduce
    154 AGE.
    155 
    156 To document these changes, please try to use ``@since`` annotations in the API
    157 specifications to explain the MAJOR revision when a feature became available,
    158 but most importantly use ``@deprecated X`` annotations to indicate that an API
    159 was deprecated and will be removed once MAJOR minus AGE is above X. When using
    160 an API, use the ``/config`` endpoints to check for compatibility and show a
    161 warning if the version(s) you support and the version(s) offered by the server
    162 are incompatible.
    163 
    164 
    165 Tagging and Package Versioning
    166 ------------------------------
    167 
    168 Release tags are of the form ``v${major}.${minor}.${patch}``.  Release tags *should* be
    169 annotated git tags.
    170 
    171 We usually consider Debian packaging files (in ``debian/``) to be part of a release.
    172 When only the Debian packaging files need to be changed, there are two options:
    173 
    174 * Make a new patch release (``v${major}.${minor}.${patch+1}``)
    175 * Make a Debian release:
    176 
    177   * Debian version now includes a revision: ``${major}.${minor}.${patch}-${debrevision}``
    178   * The tag is Debian-specific: ``debian-${major}.${minor}.${patch}-${debrevision}``
    179 
    180 All source repos *should* include a ``contrib/bump`` script that automates bumping the\
    181 version in all relevant source and packaging files.
    182 In the future, we might add an option to the script to only release a packaging bump.
    183 Right now, that process is manual.
    184 
    185 We support tagged and published pre-release versions via tags of the form ``v${major}.${minor}.${patch}-dev.${n}``.
    186 The corresponding Debian version must be ``${major}.${minor}.${patch}~dev${n}``.
    187 
    188 Nightly Debian packages should follow the `Debian conventions <https://wiki.debian.org/Versioning>`__ of ``{upcoming_version}~git{date}.{hash}-{revision}``.
    189 
    190 Testing Tools
    191 -------------
    192 
    193 For full ``make check`` support, install these programs:
    194 
    195 - `jq <https://github.com/stedolan/jq>`__
    196 - `curl <http://curl.haxx.se>`__
    197 - `faketime <https://github.com/wolfcw/libfaketime>`__
    198 
    199 The ``make check`` should be able to function without them, but
    200 their presence permits some tests to run that would otherwise be skipped.
    201 
    202 Manual Testing Database Reset
    203 -----------------------------
    204 
    205 Sometimes ``make check`` will fail with some kind of database (SQL)
    206 error, perhaps with a message like ``OBJECT does not exist`` in the
    207 ``test-suite.log`` file, where ``OBJECT`` is the name of a table or function.
    208 In that case, it may be necessary to reset the ``talercheck`` database
    209 with the commands:
    210 
    211 .. code-block:: console
    212 
    213    $ dropdb talercheck
    214    $ createdb talercheck
    215 
    216 This is because, at the moment, there is no support for
    217 doing these steps automatically in the ``make check`` flow.
    218 
    219 (If ``make check`` still fails after the reset, file a bug report as usual.)
    220 
    221 Bug Tracking
    222 ------------
    223 
    224 Bug tracking is done with Mantis (https://www.mantisbt.org/).  The bug tracker
    225 is available at `<https://bugs.taler.net>`_. A registration on the Web site is
    226 needed in order to use the bug tracker, only read access is granted without a
    227 login.
    228 
    229 We use the following conventions for the bug states:
    230 
    231 * NEW: Incoming bugs are in 'new' so that management (or developers)
    232   can easily identify those that need to be checked (report correct?
    233   something we want to fix?), prioritized and targeted for releases.
    234   "NEW" bugs are never assigned to a developer.
    235 
    236 * FEEDBACK: When blocked on feedback from reporter or other developer.
    237   Assigned to other developer (but cannot be assigned to reporter,
    238   in this case MAY remain associated with the developer who expects
    239   the feedback). If a bug is on feedback, it automatically should be
    240   considered to be high-priority to give the feedback (as it is
    241   blocking someone else!).
    242 
    243 * ACKNOWLEDGED: The bug has been reviewed, but no decision about
    244   what action to take has been made yet. Should not be worked on
    245   until management (or a developer) comes up with a plan.
    246   "ACKNOWLEDGED" bugs should NOT be assigned to a developer.
    247 
    248 * CONFIRMED: This is a real issue that should be worked on, but
    249   is not yet actively worked on. If working on this bug requires
    250   other bugs to be fixed first, they should be added as
    251   child-bugs (via relationships). Developers are always welcome
    252   to self-assign bugs that are "CONFIRMED" if they start to work
    253   on a bug.  "CONFIRMED" bugs should NOT be assigned to a developer.
    254 
    255 * ASSIGNED: The specific developer the bug is assigned to is
    256   **actively** working on the issue. Developers should strive to
    257   not have more than 5-10 bugs assigned to them at any time.
    258   Only having one assigned to you is totally OK!
    259   Developers should aggressively un-assign bugs that they are
    260   blocked on, cannot make progress on, or are no longer actively
    261   working on (but of course, better *resolve* them before
    262   moving on if possible). If the bug remains open, it probably
    263   should go back to "CONFIRMED" or "ACKNOWLEDGED".
    264 
    265 * RESOLVED: The bug has been fixed in Git.
    266 
    267 * CLOSED: An official release was made with the fix in it.
    268 
    269 When developers want to keep an eye on certain bugs, they should
    270 **monitor** them. Multiple developers can be monitoring a bug, but
    271 it can only be assigned to one.  Developers should also keep an
    272 eye on the roadmap (by release), bug categories they care about,
    273 and of course priorities / severities.
    274 
    275 We use **tags** to categorize bugs. Common tags that also imply
    276 some urgency include (in alphabetical order):
    277 
    278 * accounting: issues required for accounting (such as taxes by merchants)
    279 * compliance: issues related to regulatory compliance
    280 * $CUSTOMER: issues requested by a particular customer
    281 * performance: performance problems or ideas for improvement
    282 * security: security issues (including planned improvements to security)
    283 * UX: user experience issues
    284 
    285 These tags **should** be attached to "NEW" bugs if they apply.
    286 
    287 
    288 Code Repositories
    289 -----------------
    290 
    291 Taler code is versioned with Git. For those users without write access, all the
    292 codebases are found at the following URL:
    293 
    294 .. code-block:: none
    295 
    296    git://git.taler.net/<repository>
    297 
    298 A complete list of all the existing repositories is currently found at
    299 `<https://git.taler.net/>`_.
    300 
    301 
    302 Committing code
    303 ---------------
    304 
    305 Before you can obtain Git write access, you must sign the copyright
    306 agreement. As we collaborate closely with GNUnet, we use their
    307 copyright agreement -- with the understanding that your contributions
    308 to GNU Taler are included in the assignment.  You can find the
    309 agreement on the `GNUnet site <https://gnunet.org/en/copyright.html>`_.
    310 Please sign and mail it to Christian Grothoff as he currently collects
    311 all the documents for GNUnet e.V.
    312 
    313 To obtain Git access, you need to send us your SSH public key. Most core
    314 team members have administrative Git access, so simply contact whoever
    315 is your primary point of contact so far. You can
    316 find instructions on how to generate an SSH key
    317 in the `Git book <https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key>`_.
    318 If you have been granted write access, you first of all must change the URL of
    319 the respective repository to:
    320 
    321 .. code-block:: none
    322 
    323    ssh://git@git.taler.net/<repository>
    324 
    325 For an existing checkout, this can be done by editing the ``.git/config`` file.
    326 
    327 The server is configured to reject all commits that have not been signed with
    328 GnuPG. If you do not yet have a GnuPG key, you must create one, as explained
    329 in the `GNU Privacy Handbook <https://www.gnupg.org/gph/en/manual/c14.html>`_.
    330 You do not need to share the respective public key with us to make commits.
    331 However, we recommend that you upload it to key servers, put it on your
    332 business card and personally meet with other GNU hackers to have it signed
    333 such that others can verify your commits later.
    334 
    335 To sign all commits, you should run
    336 
    337 .. code-block:: console
    338 
    339    $ git config --global commit.gpgsign true
    340 
    341 You can also sign individual commits only by adding the ``-S`` option to the
    342 ``git commit`` command. If you accidentally already made commits but forgot
    343 to sign them, you can retroactively add signatures using:
    344 
    345 .. code-block:: console
    346 
    347    $ git rebase -S
    348 
    349 
    350 Whether you commit to a personal branch (recommended: ``dev/$USER/...``),
    351 a feature branch or to ``master`` should
    352 depend on your level of comfort and the nature of the change.  As a general
    353 rule, the code in ``master`` must always build and tests should always pass, at
    354 least on your own system. However, we all make mistakes and you should expect
    355 to receive friendly reminders if your change did not live up to this simple
    356 standard.  We plan to move to a system where the CI guarantees this invariant
    357 in the future.
    358 
    359 In order to keep a linear and clean commits history, we advise to avoid
    360 merge commits and instead always rebase your changes before pushing to
    361 the ``master`` branch.  If you commit and later find out that new commits were
    362 pushed, the following command will pull the new commits and rebase yours
    363 on top of them.
    364 
    365 .. code-block:: console
    366 
    367    # -S instructs Git to (re)sign your commits
    368    $ git pull --rebase -S
    369 
    370 
    371 
    372 Observing changes
    373 -----------------
    374 
    375 Every commit to the ``master`` branch of any of our public repositories
    376 (and almost all are public) is automatically sent to the
    377 gnunet-svn@gnu.org mailinglist.  That list is for Git commits only,
    378 and must not be used for discussions. It also carries commits from
    379 our main dependencies, namely GNUnet and GNU libmicrohttpd.  While
    380 it can be high volume, the lists is a good way to follow overall
    381 development.
    382 
    383 
    384 Code generator usage policy
    385 ---------------------------
    386 
    387 We do neither encourage nor discourage the use of tools for code generation.
    388 It is up to the individual developer to decide if a tool is acceptable for
    389 a particular task. But of course, we do encourage you to use FLOSS tools
    390 and we MUST NOT become dependent on non-free software! That said, if you
    391 use tools, you must document their use and in particular satisfy the
    392 `NLnet policy on the use of "AI" <https://nlnet.nl/news/2025/20250829-policy-on-use-of-AI.html>`__.
    393 
    394 Specifically, we ask developers to always put generated code into a *separate*
    395 Git commit and to include the full prompt in the commit message. Naturally,
    396 you may clean up the code generator's output, but then you should do so in
    397 separate Git commits (and of course only merge into master/stable after the
    398 clean up is complete). But do preserve (not squash!) the commit with the
    399 generated code so that it remains documented what the prompts were and which
    400 code is generated.  This will go a long way to keep code auditors sane!
    401 
    402 
    403 Communication
    404 -------------
    405 
    406 For public discussions we use the taler@gnu.org mailinglist.  All developers
    407 should subscribe to the low-volume Taler mailinglist.  There are separate
    408 low-volume mailinglists for gnunet-developers (@gnu.org) and for libmicrohttpd
    409 (@gnu.org).  For internal discussions we use https://mattermost.taler.net/
    410 (invitation only, but also archived).
    411 
    412 
    413 What to put in bootstrap
    414 ------------------------
    415 
    416 Each repository has a ``bootstrap`` script, which contains commands for the
    417 developer to run after a repository checkout (i.e., after ``git clone`` or
    418 ``git pull``).
    419 Typically, this updates and initializes submodules, prepares the tool chain,
    420 and runs ``autoreconf``.
    421 The last step generates the ``configure`` script, whether for immediate use or
    422 for inclusion in the distribution tarball.
    423 
    424 One common submodule is ``contrib/gana``, which pulls from the
    425 `GNUnet GANA repository <https://git.gnunet.org/gana.git/>`__.
    426 For example, in the
    427 `Taler exchange repository <https://git.taler.net/exchange.git>`__,
    428 the bootstrap script eventually runs the ``git submodule update --init`` command
    429 early on, and later runs script ``./contrib/gana-generate.sh``, which
    430 generates files such as ``src/include/taler_signatures.h``.
    431 
    432 Thus, to update that file, you need to:
    433 
    434 - (in GANA repo) Find a suitable (unused) name and number for the Signature
    435   Purposes database.
    436 
    437 - Add it to GANA, in ``gnunet-signatures/registry.rec``.
    438   (You can check for uniqueness with the ``recfix`` utility.)
    439 
    440 - Commit the change, and push it to the GANA Git repo.
    441 
    442 - (in Taler Repo) Run the ``contrib/gana-latest.sh`` script.
    443 
    444 - Bootstrap, configure, do ``make install``, ``make check``, etc.
    445   (Basically, make sure the change does not break anything.)
    446 
    447 - Commit the submodule change, and push it to the Taler exchange Git repo.
    448 
    449 A similar procedure is required for other databases in GANA.
    450 See file ``README`` in the various directories for specific instructions.
    451 
    452 
    453 Debian and Ubuntu Repositories
    454 ==============================
    455 
    456 We package our software for Debian and Ubuntu.
    457 
    458 Nightly Repositories
    459 --------------------
    460 
    461 To try the latest, unstable and untested versions of packages,
    462 you can add the nightly package sources.
    463 
    464 .. code-block:: shell-session
    465 
    466    # For Debian (trixie)
    467    $ curl -sS https://deb.taler.net/apt-nightly/taler-trixie-ci.sources \
    468      | tee /etc/apt/sources.list.d/taler-trixie-nightly.sources
    469 
    470 
    471 Taler Deployment on gv.taler.net
    472 ================================
    473 
    474 This section describes the GNU Taler deployment on ``gv.taler.net``.  ``gv``
    475 is our server at BFH. It hosts the Git repositories, Web sites, CI and other
    476 services.  Developers can receive an SSH account and e-mail alias for the
    477 system, you should contact Javier, Christian or Florian.  As with Git, ask
    478 your primary team contact for shell access if you think you need it.
    479 
    480 
    481 DNS
    482 ---
    483 
    484 DNS records for taler.net are controlled by the GNU Taler maintainers,
    485 specifically Christian and Florian, and our system administrator, Javier. If
    486 you need a sub-domain to be added, please contact one of them.
    487 
    488 
    489 User Acccounts
    490 --------------
    491 
    492 On ``gv.taler.net``, there are three system users that are set up to
    493 serve Taler on the Internet:
    494 
    495 -  ``head``: serves ``*.head.taler.net`` and gets automatically
    496    built by Buildbot every 2 hours from the ``sandcastle-ng.git``.
    497    Master key may be reset occasionally
    498 
    499 -  ``taler-test``: serves ``*.test.taler.net`` and does *NOT* get
    500    automatically built, and runs more recent tags and/or unreleased
    501    versions of Taler components. Master key may be reset
    502    occasionally.
    503 
    504 - ``demo``: serves ``*.demo.taler.net``.  Never automatically built.
    505   Master key is retained.
    506 
    507 Demo Upgrade Procedure
    508 ======================
    509 
    510 #. Login as the ``demo`` user on ``gv.taler.net``.
    511 #. Pull the latest ``sandcastle-ng.git`` code in checkout at ``$HOME/sandcastle-ng``.
    512 #. Run ``systemctl --user restart container-taler-sandcastle-demo.service``
    513 #. Refer to the sandcastle-ng README (https://git.taler.net/sandcastle-ng.git/about/)
    514    for more info.
    515 
    516 
    517 Upgrading the ``demo`` environment should be done with care, and ideally be
    518 coordinated on the mailing list before.  It is our goal for ``demo`` to always
    519 run a "working version" that is compatible with various published wallets.
    520 Please use the :doc:`demo upgrade checklist <../checklists/checklist-demo-upgrade>` to make
    521 sure everything is working.
    522 Nginx is already configured to reach the services as exported by the user unit.
    523 
    524 
    525 Tagging components
    526 ------------------
    527 
    528 All Taler components must be tagged with git before they are deployed on the
    529 ``demo`` environment, using a tag of the following form:
    530 
    531 .. code-block:: none
    532 
    533   demo-YYYY-MM-DD-SS
    534   YYYY = year
    535   MM = month
    536   DD = day
    537   SS = serial
    538 
    539 Environments and Builders on taler.net
    540 ======================================
    541 
    542 Buildbot implementation
    543 -----------------------
    544 
    545 GNU Taler uses a buildbot implementation (front end at https://buildbot.taler.net) to manage continuous integration.  Buildbot documentation is at https://docs.buildbot.net/.
    546 
    547 Here are some highlights:
    548 
    549 - The WORKER is the config that that lives on a shell account on a localhost (taler.net), where this host has buildbot-worker installed.  The WORKER executes the commands that perform all end-functions of buildbot.
    550 
    551 - The WORKER running buildbot-worker receives these commands by authenticating and communicating with the buildbot server using parameters that were specified when the worker was created in that shell account with the ``buildbot-worker`` command.
    552 
    553 - The buildbot server's master.cfg file contains FACTORY declarations which specify the commands that the WORKER will run on localhost.
    554 
    555 - The FACTORY is tied to the WORKER in master.cfg by a BUILDER.
    556 
    557 - The master.cfg also allows for SCHEDULER that defines how and when the BUILDER is executed.
    558 
    559 - Our master.cfg file is checked into git, and then periodically updated on a particular account on taler.net (ask Christian for access if needed).  Do not edit this file directly/locally on taler.net, but check changes into Git.
    560 
    561 
    562 Best Practices:
    563 
    564 - When creating a new WORKER in the ``master.cfg`` file, leave a comment specifying the server and user account that this WORKER is called from.  (At this time, taler.net is the only server used by this implementation, but it's still good practice.)
    565 
    566 - Create a worker from a shell account with this command: ``buildbot-worker create-worker <workername> localhost <username> <password>``
    567 
    568 Then make sure there is a WORKER defined in master.cfg like: ``worker.Worker("<username>", "<password>")``
    569 
    570 Test builder
    571 ------------
    572 
    573 This builder (``test-builder``) compiles and starts every Taler component.
    574 The associated worker is run by the ``taler-test`` Gv user, via the SystemD
    575 unit ``buildbot-worker-taler``.  The following commands start/stop/restart
    576 the worker:
    577 
    578 .. code-block::
    579 
    580    systemctl --user start buildbot-worker-taler
    581    systemctl --user stop buildbot-worker-taler
    582    systemctl --user restart buildbot-worker-taler
    583 
    584 .. note::
    585   the mentioned unit file can be found at ``deployment.git/systemd-services/``
    586 
    587 Wallet builder
    588 --------------
    589 
    590 This builder (``wallet-builder``) compiles every Taler component
    591 and runs the wallet integration tests.  The associated worker is
    592 run by the ``walletbuilder`` Gv user, via the SystemD unit ``buildbot-worker-wallet``.
    593 The following commands start/stop/restart the worker:
    594 
    595 .. code-block::
    596 
    597    systemctl --user start buildbot-worker-wallet
    598    systemctl --user stop buildbot-worker-wallet
    599    systemctl --user restart buildbot-worker-wallet
    600 
    601 .. note::
    602   the mentioned unit file can be found at ``deployment.git/systemd-services/``
    603 
    604 Documentation Builder
    605 ---------------------
    606 
    607 All the Taler documentation is built by the user ``docbuilder`` that
    608 runs a Buildbot worker.  The following commands set the ``docbuilder`` up,
    609 starting with an empty home directory.
    610 
    611 .. code-block:: console
    612 
    613   # Log-in as the 'docbuilder' user.
    614 
    615   $ cd $HOME
    616   $ git clone git://git.taler.net/deployment
    617   $ ./deployment/bootstrap-docbuilder
    618 
    619   # If the previous step worked, the setup is
    620   # complete and the Buildbot worker can be started.
    621 
    622   $ buildbot-worker start worker/
    623 
    624 
    625 Website Builder
    626 ---------------
    627 
    628 
    629 Taler Websites, ``www.taler.net`` and ``stage.taler.net``, are built by the
    630 user ``taler-websites`` by the means of a Buildbot worker.  The following
    631 commands set the ``taler-websites`` up, starting with an empty home directory.
    632 
    633 .. code-block:: console
    634 
    635   # Log-in as the 'taler-websites' user.
    636 
    637   $ cd $HOME
    638   $ git clone git://git.taler.net/deployment
    639   $ ./deployment/bootstrap-sitesbuilder
    640 
    641   # If the previous step worked, the setup is
    642   # complete and the Buildbot worker can be started.
    643 
    644   $ buildbot-worker start worker/
    645 
    646 
    647 Code coverage
    648 -------------
    649 
    650 Code coverage tests are run by the ``lcovworker`` user, and are also driven
    651 by Buildbot.
    652 
    653 .. code-block:: console
    654 
    655   # Log-in as the 'lcovworker' user.
    656 
    657   $ cd $HOME
    658   $ git clone git://git.taler.net/deployment
    659   $ ./deployment/bootstrap-taler lcov
    660 
    661   # If the previous step worked, the setup is
    662   # complete and the Buildbot worker can be started.
    663 
    664   $ buildbot-worker start worker/
    665 
    666 The results are then published at ``https://lcov.taler.net/``.
    667 
    668 Producing auditor reports
    669 -------------------------
    670 
    671 Both 'test' and 'demo' setups get their auditor reports compiled
    672 by a Buildbot worker.  The following steps get the reports compiler
    673 prepared.
    674 
    675 .. code-block:: console
    676 
    677   # Log-in as <env>-auditor, with <env> being either 'test' or 'demo'
    678 
    679   $ git clone git://git.taler.net/deployment
    680   $ ./deployment/buildbot/bootstrap-scripts/prepare-auditorreporter <env>
    681 
    682   # If the previous steps worked, then it should suffice to start
    683   # the worker, with:
    684 
    685   $ buildbot-worker start worker/
    686 
    687 
    688 .. _DatabaseVersioning:
    689 
    690 Database schema versioning
    691 --------------------------
    692 
    693 The PostgreSQL databases of the exchange and the auditor are versioned.
    694 See the ``versioning.sql`` file in the respective directory for documentation.
    695 
    696 Every set of changes to the database schema must be stored in a new
    697 versioned SQL script. The scripts must have contiguous numbers. After
    698 any release (or version being deployed to a production or staging
    699 environment), existing scripts MUST be immutable.
    700 
    701 Developers and operators MUST NOT make changes to database schema
    702 outside of this versioning.  All tables of a GNU Taler component should live in their own schema.
    703 
    704 
    705 QA Plans
    706 ========
    707 
    708 .. include:: ../checklists/qa-1.0.rst
    709 
    710 
    711 Releases
    712 ========
    713 
    714 .. include:: ../checklists/checklist-release.rst
    715 
    716 Release Process
    717 ---------------
    718 
    719 This document describes the process for releasing a new version of the
    720 various Taler components to the official GNU mirrors.
    721 
    722 The following components are published on the GNU mirrors
    723 
    724 -  taler-exchange (exchange.git)
    725 -  taler-merchant (merchant.git)
    726 -  sync (sync.git)
    727 -  taler-mdb (taler-mdb.git)
    728 -  libeufin (libeufin.git)
    729 -  challenger (challenger.git)
    730 -  wallet-core (wallet-core.git)
    731 
    732 Tagging
    733 -------
    734 
    735 Tag releases with an **annotated** commit, like
    736 
    737 .. code-block:: console
    738 
    739    $ git tag -a v0.1.0 -m "Official release v0.1.0"
    740    $ git push origin v0.1.0
    741 
    742 
    743 Database for tests
    744 ------------------
    745 
    746 For tests in the exchange and merchant to run, make sure that a database
    747 *talercheck* is accessible by *$USER*. Otherwise tests involving the
    748 database logic are skipped.
    749 
    750 .. include:: ../frags/db-stores-sensitive-data.rst
    751 
    752 Exchange, merchant
    753 ------------------
    754 
    755 Set the version in ``configure.ac``. The commit being tagged should be
    756 the change of the version.
    757 
    758 Tag the current GANA version that works with the exchange and merchant and
    759 checkout that tag of gana.git (instead of master). Otherwise, if there are
    760 incompatible changes in GANA (like removed symbols), old builds could break.
    761 
    762 Update the Texinfo documentation using the files from docs.git:
    763 
    764 .. code-block:: console
    765 
    766    # Get the latest documentation repository
    767    $ cd $GIT/docs
    768    $ git pull
    769    $ make texinfo
    770    # The *.texi files are now in _build/texinfo
    771    #
    772    # This checks out the prebuilt branch in the prebuilt directory
    773    $ git worktree add prebuilt prebuilt
    774    $ cd prebuilt
    775    # Copy the pre-built documentation into the prebuilt directory
    776    $ cp -r ../_build/texinfo .
    777    # Push and commit to branch
    778    $ git commit -a -S -m "updating texinfo"
    779    $ git status
    780    # Verify that all files that should be tracked are tracked,
    781    # new files will have to be added to the Makefile.am in
    782    # exchange.git as well!
    783    $ git push
    784    # Remember $REVISION of commit
    785    #
    786    # Go to exchange
    787    $ cd $GIT/exchange/doc/prebuilt
    788    # Update submodule to point to latest commit
    789    $ git checkout $REVISION
    790 
    791 Finally, the Automake ``Makefile.am`` files may have to be adjusted to
    792 include new ``*.texi`` files or images.
    793 
    794 For bootstrap, you will need to install
    795 `GNU Recutils <https://www.gnu.org/software/recutils/>`_.
    796 
    797 For the exchange test cases to pass, ``make install`` must be run first.
    798 Without it, test cases will fail because plugins can't be located.
    799 
    800 .. code-block:: console
    801 
    802    $ ./bootstrap
    803    $ ./configure # add required options for your system
    804    $ make dist
    805    $ tar -xf taler-$COMPONENT-$VERSION.tar.gz
    806    $ cd taler-$COMPONENT-$VERSION
    807    $ make install check
    808 
    809 Wallet WebExtension
    810 -------------------
    811 
    812 The version of the wallet is in *manifest.json*. The ``version_name``
    813 should be adjusted, and *version* should be increased independently on
    814 every upload to the WebStore.
    815 
    816 .. code-block:: console
    817 
    818    $ ./configure
    819    $ make dist
    820 
    821 Upload to GNU mirrors
    822 ---------------------
    823 
    824 See https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads
    825 
    826 Directive file:
    827 
    828 .. code-block:: none
    829 
    830    version: 1.2
    831    directory: taler
    832    filename: taler-exchange-0.1.0.tar.gz
    833    symlink: taler-exchange-0.1.0.tar.gz taler-exchange-latest.tar.gz
    834 
    835 Upload the files in **binary mode** to the ftp servers.
    836 
    837 
    838 Creating Debian packages
    839 ------------------------
    840 
    841 Our general setup is based on
    842 https://wiki.debian.org/DebianRepository/SetupWithReprepro
    843 
    844 First, update at least the version of the Debian package in
    845 debian/changelog, and then run:
    846 
    847 .. code-block:: bash
    848 
    849   $ dpkg-buildpackage -rfakeroot -b -uc -us
    850 
    851 in the respective source directory (GNUnet, exchange, merchant) to create the
    852 ``.deb`` files. Note that they will be created in the parent directory.  This
    853 can be done on gv.taler.net, or on another (secure) machine.
    854 Actual release builds should be done via the Docker images
    855 that can be found in ``deployment.git`` under packaging.
    856 
    857 On ``gv``, we use the ``aptbuilder`` user to manage the reprepro repository.
    858 
    859 Next, the ``*.deb`` files should be copied to gv.taler.net, say to
    860 ``/home/aptbuilder/incoming``.  Then, run
    861 
    862 .. code-block:: bash
    863 
    864   # cd /home/aptbuilder/apt
    865   # reprepro includedeb bullseye ~/incoming/*.deb
    866 
    867 to import all Debian files from ``~/incoming/`` into the ``bullseye``
    868 distribution.  If Debian packages were build against other distributions,
    869 reprepro may need to be first configured for those and the import command
    870 updated accordingly.
    871 
    872 Finally, make sure to clean up ``~/incoming/`` (by deleting the
    873 now imported ``*.deb`` files).
    874 
    875 
    876 
    877 Continuous integration
    878 ======================
    879 
    880 CI is done with Buildbot (https://buildbot.net/), and builds are
    881 triggered by the means of Git hooks. The results are published at
    882 https://buildbot.taler.net/ .
    883 
    884 In order to avoid downtimes, CI uses a "blue/green" deployment
    885 technique. In detail, there are two users building code on the system,
    886 the "green" and the "blue" user; and at any given time, one is running
    887 Taler services and the other one is either building the code or waiting
    888 for that.
    889 
    890 There is also the possibility to trigger builds manually, but this is
    891 only reserved to "admin" users.
    892 
    893 
    894 Internationalisation
    895 ====================
    896 
    897 Internationalisation (a.k.a "translation") is handled using text-based
    898 localization files named PO (Portable Object) holding pairs of original and
    899 translated strings.
    900 
    901 .. include:: dictionary.rst
    902 
    903 
    904 iOS Apps
    905 ========
    906 
    907 .. _Build-iOS-from-source:
    908 
    909 Building Taler Wallet for iOS from source
    910 -----------------------------------------
    911 
    912 The GNU Taler Wallet iOS app is in
    913 `the official Git repository <https://git.taler.net/taler-ios.git>`__.
    914 
    915 Compatibility
    916 ^^^^^^^^^^^^^
    917 
    918 The minimum version of iOS supported is 15.0.
    919 This app runs on all iPhone models at least as new as the iPhone 6S.
    920 
    921 
    922 Building
    923 ^^^^^^^^
    924 
    925 Before building the iOS wallet, you must first checkout the
    926 `quickjs-tart repo <https://git.taler.net/quickjs-tart.git>`__
    927 and the
    928 `wallet-core repo <https://git.taler.net/wallet-core.git>`__.
    929 
    930 Have all 3 local repos (wallet-core, quickjs-tart, and this one) adjacent at
    931 the same level (e.g. in a "GNU_Taler" folder)
    932 Taler.xcworkspace expects the QuickJS framework sub-project to be at
    933 ``../quickjs-tart/QuickJS-rt.xcodeproj``.
    934 
    935 Build wallet-core first:
    936 
    937 .. code-block:: shell-session
    938 
    939   $ cd wallet-core
    940   $ make embedded
    941   $ open packages/taler-wallet-embedded/dist
    942 
    943 then drag or move its product "taler-wallet-core-qjs.mjs"
    944 into your quickjs-tart folder right at the top level.
    945 
    946 Open Taler.xcworkspace, and set scheme / target to Taler_Wallet. Build&run...
    947 
    948 Don't open QuickJS-rt.xcodeproj or TalerWallet.xcodeproj and build anything
    949 there - all needed libraries and frameworks will be built automatically from
    950 Taler.xcworkspace.
    951 
    952 
    953 Android Apps
    954 ============
    955 
    956 Android App Nightly Builds
    957 --------------------------
    958 
    959 There are currently three Android apps in
    960 `the official Git repository <https://git.taler.net/taler-android.git>`__:
    961 
    962 * Wallet
    963   [`CI <https://git.taler.net/taler-android.git/tree/wallet/.gitlab-ci.yml>`__]
    964 * Merchant PoS Terminal
    965   [`CI <https://git.taler.net/taler-android.git/tree/merchant-terminal/.gitlab-ci.yml>`__]
    966 * Cashier
    967   [`CI <https://git.taler.net/taler-android.git/tree/cashier/.gitlab-ci.yml>`__]
    968 
    969 Their git repositories are `mirrored at Gitlab <https://gitlab.com/gnu-taler/taler-android>`__
    970 to utilize their CI
    971 and `F-Droid <https://f-droid.org>`_'s Gitlab integration
    972 to `publish automatic nightly builds <https://f-droid.org/docs/Publishing_Nightly_Builds/>`_
    973 for each change on the ``master`` branch.
    974 
    975 All three apps publish their builds to the same F-Droid nightly repository
    976 (which is stored as a git repository):
    977 https://gitlab.com/gnu-taler/fdroid-repo-nightly
    978 
    979 You can download the APK files directly from that repository
    980 or add it to the F-Droid app for automatic updates
    981 by clicking the following link (on the phone that has F-Droid installed).
    982 
    983     `GNU Taler Nightly F-Droid Repository <fdroidrepos://gnu-taler.gitlab.io/fdroid-repo-nightly/fdroid/repo?fingerprint=55F8A24F97FAB7B0960016AF393B7E57E7A0B13C2D2D36BAC50E1205923A7843>`_
    984 
    985 .. note::
    986     Nightly apps can be installed alongside official releases
    987     and thus are meant **only for testing purposes**.
    988     Use at your own risk!
    989 
    990 .. _Build-apps-from-source:
    991 
    992 Building apps from source
    993 -------------------------
    994 
    995 Note that this guide is different from other guides for building Android apps,
    996 because it does not require you to run non-free software.
    997 It uses the Merchant PoS Terminal as an example, but works as well for the other apps
    998 if you replace ``merchant-terminal`` with ``wallet`` or ``cashier``.
    999 
   1000 First, ensure that you have the required dependencies installed:
   1001 
   1002 * Java Development Kit 8 or higher (default-jdk-headless)
   1003 * git
   1004 * unzip
   1005 
   1006 Then you can get the app's source code using git:
   1007 
   1008 .. code-block:: console
   1009 
   1010   # Start by cloning the Android git repository
   1011   $ git clone https://git.taler.net/taler-android.git
   1012 
   1013   # Change into the directory of the cloned repository
   1014   $ cd taler-android
   1015 
   1016   # Find out which Android SDK version you will need
   1017   $ grep -i compileSdkVersion merchant-terminal/build.gradle
   1018 
   1019 The last command will return something like ``compileSdkVersion 29``.
   1020 So visit the `Android Rebuilds <http://android-rebuilds.beuc.net/>`_ project
   1021 and look for that version of the Android SDK there.
   1022 If the SDK version is not yet available as a free rebuild,
   1023 you can try to lower the ``compileSdkVersion`` in the app's ``merchant-terminal/build.gradle`` file.
   1024 Note that this might break things
   1025 or require you to also lower other versions such as ``targetSdkVersion``.
   1026 
   1027 In our example, the version is ``29`` which is available,
   1028 so download the "SDK Platform" package of "Android 10.0.0 (API 29)"
   1029 and unpack it:
   1030 
   1031 .. code-block:: console
   1032 
   1033   # Change into the directory that contains your downloaded SDK
   1034   $ cd $HOME
   1035 
   1036   # Unpack/extract the Android SDK
   1037   $ unzip android-sdk_eng.10.0.0_r14_linux-x86.zip
   1038 
   1039   # Tell the build system where to find the SDK
   1040   $ export ANDROID_SDK_ROOT="$HOME/android-sdk_eng.10.0.0_r14_linux-x86"
   1041 
   1042   # Change into the directory of the cloned repository
   1043   $ cd taler-android
   1044 
   1045   # Build the merchant-terminal app
   1046   $ ./gradlew :merchant-terminal:assembleRelease
   1047 
   1048 If you get an error message complaining about build-tools
   1049 
   1050     > Failed to install the following Android SDK packages as some licences have not been accepted.
   1051          build-tools;29.0.3 Android SDK Build-Tools 29.0.3
   1052 
   1053 you can try changing the ``buildToolsVersion`` in the app's ``merchant-terminal/build.gradle`` file
   1054 to the latest "Android SDK build tools" version supported by the Android Rebuilds project.
   1055 
   1056 After the build finished successfully,
   1057 you will find your APK in ``merchant-terminal/build/outputs/apk/release/``.
   1058 
   1059 Update translations
   1060 -------------------
   1061 
   1062 Translations are managed with Taler's weblate instance:
   1063 https://weblate.taler.net/projects/gnu-taler/
   1064 
   1065 To update translations, enter the taler-android git repository
   1066 and ensure that the weblate remote exists:
   1067 
   1068 .. code-block:: console
   1069 
   1070   $ git config -l | grep weblate
   1071 
   1072 If it does not yet exist (empty output), you can add it like this:
   1073 
   1074 .. code-block:: console
   1075 
   1076   $ git remote add weblate https://weblate.taler.net/git/gnu-taler/wallet-android/
   1077 
   1078 Then you can merge in translations commit from the weblate remote:
   1079 
   1080 .. code-block:: console
   1081 
   1082   # ensure you have latest version
   1083   $ git fetch weblate
   1084 
   1085   # merge in translation commits
   1086   $ git merge weblate/master
   1087 
   1088 Afterwards, build the entire project from source and test the UI
   1089 to ensure that no erroneous translations (missing placeholders) are breaking things.
   1090 
   1091 Release process
   1092 ---------------
   1093 
   1094 After extensive testing, the code making up a new release should get a signed git tag.
   1095 The current tag format is:
   1096 
   1097 * cashier-$VERSION
   1098 * pos-$VERSION
   1099 * wallet-$VERSION (where $VERSION has a v prefix)
   1100 
   1101 .. code-block:: console
   1102 
   1103   $ git tag -s $APP-$VERSION
   1104 
   1105 F-Droid
   1106 ^^^^^^^
   1107 Nightly builds get published automatically (see above) after pushing code to the official repo.
   1108 Actual releases get picked up by F-Droid's official repository via git tags.
   1109 So ensure that all releases get tagged properly.
   1110 
   1111 Some information for F-Droid official repository debugging:
   1112 
   1113 * Wallet: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.wallet.fdroid.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.wallet.fdroid/lastbuild>`__]
   1114 * Cashier: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.cashier.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.cashier/lastbuild>`__]
   1115 * PoS: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.merchantpos.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.merchantpos/lastbuild>`__]
   1116 
   1117 Google Play
   1118 ^^^^^^^^^^^
   1119 Google Play uploads are managed via `Fastlane <https://docs.fastlane.tools/getting-started/android/setup/>`__.
   1120 Before proceeding, ensure that this is properly set up
   1121 and that you have access to the Google Play API.
   1122 
   1123 It is important to have access to the signing keys and Google Play access keys
   1124 (JSON) and to ensure that the following environment variables are set
   1125 correctly and made available to Fastlane:
   1126 
   1127 .. code-block:: bash
   1128 
   1129    TALER_KEYSTORE_PATH=
   1130    TALER_KEYSTORE_PASS=
   1131    TALER_KEYSTORE_WALLET_ALIAS=
   1132    TALER_KEYSTORE_WALLET_PASS=
   1133    TALER_KEYSTORE_POS_ALIAS=
   1134    TALER_KEYSTORE_POS_PASS=
   1135    TALER_KEYSTORE_CASHIER_ALIAS=
   1136    TALER_KEYSTORE_CASHIER_PASS=
   1137    TALER_JSON_KEY_FILE=
   1138 
   1139 To release an app, enter into its respective folder and run fastlane:
   1140 
   1141 .. code-block:: console
   1142 
   1143   $ bundle exec fastlane
   1144 
   1145 Then select the deploy option.
   1146 
   1147 All uploads are going to the beta track by default.  These can be promoted to
   1148 production later or immediately after upload if you feel daring. It is also
   1149 important to bump the version and build code with every release.
   1150 
   1151 .. _Code-coverage:
   1152 
   1153 Code Coverage
   1154 =============
   1155 
   1156 Code coverage is done with the Gcov / Lcov
   1157 (http://ltp.sourceforge.net/coverage/lcov.php) combo, and it is run
   1158 nightly (once a day) by a Buildbot worker. The coverage results are
   1159 then published at https://lcov.taler.net/ .
   1160 
   1161 
   1162 Coding Conventions
   1163 ==================
   1164 
   1165 GNU Taler is developed primarily in C, Kotlin, Python, Swift and TypeScript.
   1166 
   1167 Components written in C
   1168 -----------------------
   1169 
   1170 These are the general coding style rules for Taler.
   1171 
   1172 * Baseline rules are to follow GNU guidelines, modified or extended
   1173   by the GNUnet style: https://docs.gnunet.org/handbook/gnunet.html#Coding-style
   1174 
   1175 Naming conventions
   1176 ^^^^^^^^^^^^^^^^^^
   1177 
   1178 * include files (very similar to GNUnet):
   1179 
   1180   * if installed, must start with "``taler_``" (exception: platform.h),
   1181     and MUST live in src/include/
   1182   * if NOT installed, must NOT start with "``taler_``" and
   1183     MUST NOT live in src/include/ and
   1184     SHOULD NOT be included from outside of their own directory
   1185   * end in "_lib" for "simple" libraries
   1186   * end in "_plugin" for plugins
   1187   * end in "_service" for libraries accessing a service, i.e. the exchange
   1188 
   1189 * binaries:
   1190 
   1191   * taler-exchange-xxx: exchange programs
   1192   * taler-merchant-xxx: merchant programs (demos)
   1193   * taler-wallet-xxx: wallet programs
   1194   * plugins should be libtaler_plugin_xxx_yyy.so: plugin yyy for API xxx
   1195   * libtalerxxx: library for API xxx
   1196 
   1197 * logging
   1198 
   1199   * tools use their full name in GNUNET_log_setup
   1200     (i.e. 'taler-exchange-offline') and log using plain 'GNUNET_log'.
   1201   * pure libraries (without associated service) use 'GNUNET_log_from'
   1202     with the component set to their library name (without lib or '.so'),
   1203     which should also be their directory name (i.e. 'util')
   1204   * plugin libraries (without associated service) use 'GNUNET_log_from'
   1205     with the component set to their type and plugin name (without lib or '.so'),
   1206     which should also be their directory name (i.e. 'exchangedb-postgres')
   1207   * libraries with associated service) use 'GNUNET_log_from'
   1208     with the name of the service,  which should also be their
   1209     directory name (i.e. 'exchange')
   1210   * for tools with ``-l LOGFILE``, its absence means write logs to stderr
   1211 
   1212 * configuration
   1213 
   1214   * same rules as for GNUnet
   1215 
   1216 * exported symbols
   1217 
   1218   * must start with TALER_[SUBSYSTEMNAME]_ where SUBSYSTEMNAME
   1219     MUST match the subdirectory of src/ in which the symbol is defined
   1220   * from libtalerutil start just with ``TALER_``, without subsystemname
   1221   * if scope is ONE binary and symbols are not in a shared library,
   1222     use binary-specific prefix (such as TMH = taler-exchange-httpd) for
   1223     globals, possibly followed by the subsystem (TMH_DB_xxx).
   1224 
   1225 * structs:
   1226 
   1227   * structs that are 'packed' and do not contain pointers and are
   1228     thus suitable for hashing or similar operations are distinguished
   1229     by adding a "P" at the end of the name. (NEW)  Note that this
   1230     convention does not hold for the GNUnet-structs (yet).
   1231   * structs that are used with a purpose for signatures, additionally
   1232     get an "S" at the end of the name.
   1233 
   1234 * private (library-internal) symbols (including structs and macros)
   1235 
   1236   * must not start with ``TALER_`` or any other prefix
   1237 
   1238 * testcases
   1239 
   1240   * must be called "test_module-under-test_case-description.c"
   1241 
   1242 * performance tests
   1243 
   1244   * must be called "perf_module-under-test_case-description.c"
   1245 
   1246 Shell Scripts
   1247 -------------
   1248 
   1249 Shell scripts should be avoided if at all possible.  The only permissible uses of shell scripts
   1250 in GNU Taler are:
   1251 
   1252 * Trivial invocation of other commands.
   1253 * Scripts for compatibility (e.g. ``./configure``) that must run on
   1254   as many systems as possible.
   1255 
   1256 When shell scripts are used, they ``MUST`` begin with the following ``set`` command:
   1257 
   1258 .. code-block:: console
   1259 
   1260   # Make the shell fail on undefined variables and
   1261   # commands with non-zero exit status.
   1262   $ set -eu
   1263 
   1264 Kotlin
   1265 ------
   1266 
   1267 We so far have no specific guidelines, please follow best practices
   1268 for the language.
   1269 
   1270 
   1271 Python
   1272 ------
   1273 
   1274 Supported Python Versions
   1275 ^^^^^^^^^^^^^^^^^^^^^^^^^
   1276 
   1277 Python code should be written and built against version 3.7 of Python.
   1278 
   1279 Style
   1280 ^^^^^
   1281 
   1282 We use `yapf <https://github.com/google/yapf>`_ to reformat the
   1283 code to conform to our style instructions.
   1284 A reusable yapf style file can be found in ``build-common``,
   1285 which is intended to be used as a git submodule.
   1286 
   1287 Python for Scripting
   1288 ^^^^^^^^^^^^^^^^^^^^
   1289 
   1290 When using Python for writing small utilities, the following libraries
   1291 are useful:
   1292 
   1293 * ``click`` for argument parsing (should be preferred over argparse)
   1294 * ``pathlib`` for path manipulation (part of the standard library)
   1295 * ``subprocess`` for "shelling out" to other programs.  Prefer ``subprocess.run``
   1296   over the older APIs.
   1297 
   1298 
   1299 Swift
   1300 -----
   1301 
   1302 Please follow best practices for the language.
   1303 
   1304 
   1305 TypeScript
   1306 ----------
   1307 
   1308 Please follow best practices for the language.
   1309 
   1310 
   1311 Testing library
   1312 ===============
   1313 
   1314 This chapter is a VERY ABSTRACT description of how testing is
   1315 implemented in Taler, and in NO WAY wants to substitute the reading of
   1316 the actual source code by the user.
   1317 
   1318 In Taler, a test case is an array of ``struct TALER_TESTING_Command``,
   1319 informally referred to as ``CMD``, that is iteratively executed by the
   1320 testing interpreter. This latter is transparently initiated by the
   1321 testing library.
   1322 
   1323 However, the developer does not have to define CMDs manually, but
   1324 rather call the proper constructor provided by the library. For example,
   1325 if a CMD is supposed to test feature ``x``, then the library would
   1326 provide the ``TALER_TESTING_cmd_x ()`` constructor for it. Obviously,
   1327 each constructor has its own particular arguments that make sense to
   1328 test ``x``, and all constructors are thoroughly commented within the
   1329 source code.
   1330 
   1331 Internally, each CMD has two methods: ``run ()`` and ``cleanup ()``. The
   1332 former contains the main logic to test feature ``x``, whereas the latter
   1333 cleans the memory up after execution.
   1334 
   1335 In a test life, each CMD needs some internal state, made by values it
   1336 keeps in memory. Often, the test has to *share* those values with other
   1337 CMDs: for example, CMD1 may create some key material and CMD2 needs this
   1338 key material to encrypt data.
   1339 
   1340 The offering of internal values from CMD1 to CMD2 is made by *traits*. A
   1341 trait is a ``struct TALER_TESTING_Trait``, and each CMD contains an array
   1342 of traits, that it offers via the public trait interface to other
   1343 commands. The definition and filling of such array happens transparently
   1344 to the test developer.
   1345 
   1346 For example, the following example shows how CMD2 takes an amount object
   1347 offered by CMD1 via the trait interface.
   1348 
   1349 Note: the main interpreter and the most part of CMDs and traits are
   1350 hosted inside the exchange codebase, but nothing prevents the developer
   1351 from implementing new CMDs and traits within other codebases.
   1352 
   1353 .. code-block:: c
   1354 
   1355    /* Without loss of generality, let's consider the
   1356     * following logic to exist inside the run() method of CMD1 */
   1357    ...
   1358 
   1359    struct TALER_Amount *a;
   1360    /**
   1361     * the second argument (0) points to the first amount object offered,
   1362     * in case multiple are available.
   1363     */
   1364    if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a))
   1365      return GNUNET_SYSERR;
   1366    ...
   1367 
   1368    use(a); /* 'a' points straight into the internal state of CMD2 */
   1369 
   1370 In the Taler realm, there is also the possibility to alter the behaviour
   1371 of supposedly well-behaved components. This is needed when, for example,
   1372 we want the exchange to return some corrupted signature in order to
   1373 check if the merchant backend detects it.
   1374 
   1375 This alteration is accomplished by another service called *twister*. The
   1376 twister acts as a proxy between service A and B, and can be programmed
   1377 to tamper with the data exchanged by A and B.
   1378 
   1379 Please refer to the Twister codebase (under the ``test`` directory) in
   1380 order to see how to configure it.
   1381 
   1382 
   1383 User-Facing Terminology
   1384 =======================
   1385 
   1386 This section contains terminology that should be used and that should not be
   1387 used in the user interface and help materials.
   1388 
   1389 Terms to Avoid
   1390 --------------
   1391 
   1392 Refreshing
   1393   Refreshing is the internal technical terminology for the protocol to
   1394   give change for partially spent coins
   1395 
   1396   **Use instead**: "Obtaining change"
   1397 
   1398 Charge
   1399   Charge has two opposite meanings (charge to a credit card vs. charge a battery).
   1400   This can confuse users.
   1401 
   1402   **Use instead**: "Obtain", "Credit", "Debit", "Withdraw", "Top up"
   1403 
   1404 Coin
   1405   Coins are an internal construct, the user should never be aware that their balance
   1406   is represented by coins of different denominations.
   1407 
   1408   **Use instead**: "(Digital) Cash" or "(Wallet) Balance"
   1409 
   1410 Consumer
   1411   Has bad connotation of consumption.
   1412 
   1413   **Use instead**: Customer or user.
   1414 
   1415 Proposal
   1416   The term used to describe the process of the merchant facilitating the download
   1417   of the signed contract terms for an order.
   1418 
   1419   **Avoid**.  Generally events that relate to proposal downloads
   1420   should not be shown to normal users, only developers.  Instead, use
   1421   "communication with merchant failed" if a proposed order can't be downloaded.
   1422 
   1423 Anonymous E-Cash
   1424   Should be generally avoided, since Taler is only anonymous for
   1425   the customer. Also some people are scared of anonymity (which as
   1426   a term is also way too absolute, as anonymity is hardly ever perfect).
   1427 
   1428   **Use instead**:  "Privacy-preserving", "Privacy-friendly"
   1429 
   1430 Payment Replay
   1431   The process of proving to the merchant that the customer is entitled
   1432   to view a digital product again, as they already paid for it.
   1433 
   1434   **Use instead**:  In the event history, "re-activated digital content purchase"
   1435   could be used. (FIXME: this is still not nice.)
   1436 
   1437 Session ID
   1438   See Payment Replay.
   1439 
   1440 Order
   1441   Too ambiguous in the wallet.
   1442 
   1443   **Use instead**: Purchase
   1444 
   1445 Fulfillment URL
   1446   URL that serves the digital content that the user purchased
   1447   with their payment.  Can also be something like a donation receipt.
   1448 
   1449 Donau
   1450   Developer-internal name for the tax authority component.
   1451 
   1452   **Use instead**:  Tax authority
   1453 
   1454 Terms to Use
   1455 ------------
   1456 
   1457 Auditor
   1458   Regulatory entity that certifies exchanges and oversees their operation.
   1459 
   1460 Exchange Operator
   1461   The entity/service that gives out digital cash in exchange for some
   1462   other means of payment.
   1463 
   1464   In some contexts, using "Issuer" could also be appropriate.
   1465   When showing a balance breakdown,
   1466   we can say "100 Eur (issued by exchange.euro.taler.net)".
   1467   Sometimes we may also use the more generic term "Payment Service Provider"
   1468   when the concept of an "Exchange" is still unclear to the reader.
   1469 
   1470 Refund
   1471   A refund is given by a merchant to the customer (rather the customer's wallet)
   1472   and "undoes" a previous payment operation.
   1473 
   1474 Payment
   1475   The act of sending digital cash to a merchant to pay for an order.
   1476 
   1477 Purchase
   1478   Used to refer to the "result" of a payment, as in "view purchase".
   1479   Use sparingly, as the word doesn't fit for all payments, such as donations.
   1480 
   1481 Contract Terms
   1482   Partially machine-readable representation of the merchant's obligation after the
   1483   customer makes a payment.
   1484 
   1485 Merchant
   1486   Party that receives a payment.
   1487 
   1488 Wallet
   1489   Also "Taler Wallet".  Software component that manages the user's digital cash
   1490   and payments.
   1491 
   1492 
   1493 Developer Glossary
   1494 ==================
   1495 
   1496 This glossary is meant for developers.  It contains some terms that we usually do not
   1497 use when talking to end users or even system administrators.
   1498 
   1499 .. glossary::
   1500   :sorted:
   1501 
   1502   absolute time
   1503     method of keeping time in :term:`GNUnet` where the time is represented
   1504     as the number of microseconds since 1.1.1970 (UNIX epoch).  Called
   1505     absolute time in contrast to :term:`relative time`.
   1506 
   1507   aggregate
   1508     the :term:`exchange` combines multiple payments received by the
   1509     same :term:`merchant` into one larger :term:`wire transfer` to
   1510     the respective merchant's :term:`bank` account
   1511 
   1512   auditor
   1513     trusted third party that verifies that the :term:`exchange` is operating correctly
   1514 
   1515   bank
   1516     traditional financial service provider who offers
   1517     :term:`wire transfers <wire transfer>` between accounts
   1518 
   1519   buyer
   1520     individual in control of a Taler :term:`wallet`, usually using it to
   1521     :term:`spend` the :term:`coins <coin>` on :term:`contracts <contract>` (see also :term:`customer`).
   1522 
   1523   close
   1524     operation an :term:`exchange` performs on a :term:`reserve` that has not been
   1525     :term:`emptied <empty>` by :term:`withdraw` operations. When closing a reserve, the
   1526     exchange wires the remaining funds back to the customer, minus a :term:`fee`
   1527     for closing
   1528 
   1529   customer
   1530     individual that directs the buyer (perhaps the same individual) to make a purchase
   1531 
   1532   coin
   1533     coins are individual tokens representing a certain amount of value, also known as the :term:`denomination` of the coin
   1534 
   1535   refresh commitment
   1536     data that the wallet commits to during the :term:`melt` stage of the
   1537     :term:`refresh` protocol where it
   1538     has to prove to the :term:`exchange` that it is deriving the :term:`fresh`
   1539     coins as specified by the Taler protocol.  The commitment is verified
   1540     probabilistically (see: :term:`kappa`) during the :term:`reveal` stage.
   1541 
   1542   contract
   1543     formal agreement between :term:`merchant` and :term:`customer` specifying the
   1544     :term:`contract terms` and signed by the merchant and the :term:`coins <coin>` of the
   1545     customer
   1546 
   1547   contract terms
   1548     the individual clauses specifying what the buyer is purchasing from the
   1549     :term:`merchant`
   1550 
   1551   denomination
   1552     unit of currency, specifies both the currency and the face value of a :term:`coin`,
   1553     as well as associated fees and validity periods
   1554 
   1555   denomination key
   1556     (RSA) key used by the exchange to certify that a given :term:`coin` is valid and of a
   1557     particular :term:`denomination`
   1558 
   1559   deposit
   1560     operation by which a merchant passes coins to an exchange, expecting the
   1561     exchange to credit his bank account in the future using an
   1562     :term:`aggregate` :term:`wire transfer`
   1563 
   1564   drain
   1565     process by which an exchange operator takes the profits
   1566     (from :term:`fees <fee>`) out of the escrow account and moves them into
   1567     their regular business account
   1568 
   1569   dirty
   1570     a :term:`coin` is dirty if its public key may be known to an entity other than
   1571     the customer, thereby creating the danger of some entity being able to
   1572     link multiple transactions of coin's owner if the coin is not refreshed
   1573 
   1574   empty
   1575     a :term:`reserve` is being emptied when a :term:`wallet` is using the
   1576     reserve's private key to :term:`withdraw` coins from it. This reduces
   1577     the balance of the reserve. Once the balance reaches zero, we say that
   1578     the reserve has been (fully) emptied.  Reserves that are not emptied
   1579     (which is the normal process) are :term:`closed <close>` by the exchange.
   1580 
   1581   exchange
   1582     Taler's payment service operator.  Issues electronic coins during
   1583     withdrawal and redeems them when they are deposited by merchants
   1584 
   1585   expired
   1586     Various operations come with time limits. In particular, denomination keys
   1587     come with strict time limits for the various operations involving the
   1588     coin issued under the denomination. The most important limit is the
   1589     deposit expiration, which specifies until when wallets are allowed to
   1590     use the coin in deposit or refreshing operations. There is also a "legal"
   1591     expiration, which specifies how long the exchange keeps records beyond the
   1592     deposit expiration time.  This latter expiration matters for legal disputes
   1593     in courts and also creates an upper limit for refreshing operations on
   1594     special zombie coin
   1595 
   1596   GNUnet
   1597     Codebase of various libraries for a better Internet, some of which
   1598     GNU Taler depends upon.
   1599 
   1600   fakebank
   1601     implementation of the :term:`bank` API in memory to be used only for test
   1602     cases.
   1603 
   1604   fee
   1605     an :term:`exchange` charges various fees for its service. The different
   1606     fees are specified in the protocol. There are fees per coin for
   1607     :term:`withdrawing <withdraw>`, :term:`depositing <deposit>`, :term:`melting <melt>`, and
   1608     :term:`refunding <refund>`.  Furthermore, there are fees per wire transfer
   1609     when a :term:`reserve` is :term:`closed <close>`
   1610     and for :term:`aggregate` :term:`wire transfers <wire transfer>`
   1611     to the :term:`merchant`.
   1612 
   1613   fresh
   1614     a :term:`coin` is fresh if its public key is only known to the customer
   1615 
   1616   JSON
   1617     JavaScript Object Notation (JSON) is a
   1618     serialization format derived from the JavaScript language which is
   1619     commonly used in the Taler protocol as the payload of HTTP requests
   1620     and responses.
   1621 
   1622   kappa
   1623     security parameter used in the :term:`refresh` protocol. Defined to be 3.
   1624     The probability of successfully evading the income transparency with the
   1625     refresh protocol is 1:kappa.
   1626 
   1627   libeufin
   1628     Kotlin component that implements a regional currency bank and an
   1629     adapter to communicate via EBICS with European core banking systems.
   1630 
   1631   link
   1632     specific step in the :term:`refresh` protocol that an exchange must offer
   1633     to prevent abuse of the :term:`refresh` mechanism.  The link step is
   1634     not needed in normal operation, it just must be offered.
   1635 
   1636   master key
   1637     offline key used by the exchange to certify denomination keys and
   1638     message signing keys
   1639 
   1640   melt
   1641     step of the :term:`refresh` protocol where a :term:`dirty` :term:`coin`
   1642     is invalidated to be reborn :term:`fresh` in a subsequent
   1643     :term:`reveal` step.
   1644 
   1645   merchant
   1646     party receiving payments (usually in return for goods or services)
   1647 
   1648   message signing key
   1649      key used by the exchange to sign online messages, other than coins
   1650 
   1651   order
   1652     offer made by the merchant to a wallet; pre-cursor to
   1653     a contract where the wallet is not yet fixed. Turns
   1654     into a :term:`contract` when a wallet claims the order.
   1655 
   1656   owner
   1657     a coin is owned by the entity that knows the private key of the coin
   1658 
   1659   relative time
   1660     method of keeping time in :term:`GNUnet` where the time is represented
   1661     as a relative number of microseconds.  Thus, a relative time specifies
   1662     an offset or a duration, but not a date.  Called relative time in
   1663     contrast to :term:`absolute time`.
   1664 
   1665   recoup
   1666     Operation by which an exchange returns the value of coins affected
   1667     by a :term:`revocation <revoke>` to their :term:`owner`, either by allowing the owner to
   1668     withdraw new coins or wiring funds back to the bank account of the :term:`owner`.
   1669 
   1670   planchet
   1671     precursor data for a :term:`coin`. A planchet includes the coin's internal
   1672     secrets (coin private key, blinding factor), but lacks the RSA signature
   1673     of the :term:`exchange`.  When :term:`withdrawing <withdraw>`, a :term:`wallet`
   1674     creates and persists a planchet before asking the exchange to sign it to
   1675     get the coin.
   1676 
   1677   purchase
   1678     Refers to the overall process of negotiating a :term:`contract` and then
   1679     making a payment with :term:`coins <coin>` to a :term:`merchant`.
   1680 
   1681   privacy policy
   1682     Statement of an operator how they will protect the privacy of users.
   1683 
   1684   proof
   1685     Message that cryptographically demonstrates that a particular claim is correct.
   1686 
   1687   proposal
   1688     a list of :term:`contract terms` that has been completed and signed by the
   1689     merchant backend.
   1690 
   1691   refresh
   1692     operation by which a :term:`dirty` :term:`coin` is converted into one or more
   1693     :term:`fresh` coins.  Involves :term:`melting <melt>` the :term:`dirty` coins and
   1694     then :term:`revealing <reveal>` so-called :term:`transfer keys <transfer key>`.
   1695 
   1696   refund
   1697     operation by which a merchant steps back from the right to funds that he
   1698     obtained from a :term:`deposit` operation, giving the right to the funds back
   1699     to the customer
   1700 
   1701   refund transaction id
   1702     unique number by which a merchant identifies a :term:`refund`. Needed
   1703     as refunds can be partial and thus there could be multiple refunds for
   1704     the same :term:`purchase`.
   1705 
   1706   reserve
   1707     accounting mechanism used by the exchange to track customer funds
   1708     from incoming :term:`wire transfers <wire transfer>`.  A reserve is created whenever
   1709     a customer wires money to the exchange using a well-formed public key
   1710     in the subject.  The exchange then allows the customer's :term:`wallet`
   1711     to :term:`withdraw` up to the amount received in :term:`fresh`
   1712     :term:`coins <coin>` from the reserve, thereby emptying the reserve. If a
   1713     reserve is not emptied, the exchange will eventually :term:`close` it.
   1714 
   1715     Other definition: Funds set aside for future use; either the balance of a customer at the
   1716     exchange ready for withdrawal, or the funds kept in the exchange;s bank
   1717     account to cover obligations from coins in circulation.
   1718 
   1719   reveal
   1720     step in the :term:`refresh` protocol where some of the transfer private
   1721     keys are revealed to prove honest behavior on the part of the wallet.
   1722     In the reveal step, the exchange returns the signed :term:`fresh` coins.
   1723 
   1724   revoke
   1725     exceptional operation by which an exchange withdraws a denomination from
   1726     circulation, either because the signing key was compromised or because
   1727     the exchange is going out of operation; unspent coins of a revoked
   1728     denomination are subjected to recoup.
   1729 
   1730   sharing
   1731     users can share ownership of a :term:`coin` by sharing access to the coin&#39;s
   1732     private key, thereby allowing all co-owners to spend the coin at any
   1733     time.
   1734 
   1735   spend
   1736     operation by which a customer gives a merchant the right to deposit
   1737     coins in return for merchandise
   1738 
   1739   transfer key
   1740     special cryptographic key used in the :term:`refresh` protocol, some of which
   1741     are revealed during the :term:`reveal` step. Note that transfer keys have,
   1742     despite the name, no relationship to :term:`wire transfers <wire transfer>`.  They merely
   1743     help to transfer the value from a :term:`dirty` coin to a :term:`fresh` coin
   1744 
   1745   terms
   1746     the general terms of service of an operator, possibly including
   1747     the :term:`privacy policy`.  Not to be confused with the
   1748     :term:`contract terms` which are about the specific purchase.
   1749 
   1750   transaction
   1751     method by which ownership is exclusively transferred from one entity
   1752 
   1753   user
   1754     any individual using the Taler payment system
   1755     (see :term:`customer`, :term:`buyer`, :term:`merchant`).
   1756 
   1757   version
   1758     Taler uses various forms of versioning. There is a database
   1759     schema version (stored itself in the database, see \*-0000.sql) describing
   1760     the state of the table structure in the database of an :term:`exchange`,
   1761     :term:`auditor` or :term:`merchant`. There is a protocol
   1762     version (CURRENT:REVISION:AGE, see GNU libtool) which specifies
   1763     the network protocol spoken by an :term:`exchange` or :term:`merchant`
   1764     including backwards-compatibility. And finally there is the software
   1765     release version (MAJOR.MINOR.PATCH, see https://semver.org/) of
   1766     the respective code base.
   1767 
   1768   wallet
   1769     software running on a customer's computer; withdraws, stores and
   1770     spends coins
   1771 
   1772   WebExtension
   1773     Cross-browser API used to implement the GNU Taler wallet browser extension.
   1774 
   1775   wire gateway
   1776     API used by the exchange to talk with some real-time gross settlement system
   1777     (core banking system, blockchain) to notice inbound credits wire transfers
   1778     (during withdraw) and to trigger outbound debit wire transfers (primarily
   1779     for deposits).
   1780 
   1781   wire transfer
   1782     a wire transfer is a method of sending funds between :term:`bank` accounts
   1783 
   1784   wire transfer identifier
   1785     Subject of a wire transfer from the exchange to a merchant;
   1786     set by the aggregator to a random nonce which uniquely
   1787     identifies the transfer.
   1788 
   1789   withdraw
   1790     operation by which a :term:`wallet` can convert funds from a :term:`reserve` to
   1791     fresh coins
   1792 
   1793   zombie
   1794     :term:`coin` where the respective :term:`denomination key` is past its
   1795     :term:`deposit` :term:`expiration <expired>` time, but which is still (again) valid
   1796     for an operation because it was :term:`melted <melt>` while it was still
   1797     valid, and then later again credited during a :term:`recoup` process
   1798 
   1799 
   1800 
   1801 Developer Tools
   1802 ===============
   1803 
   1804 This section describes various internal programs to make life easier for the
   1805 developer.
   1806 
   1807 
   1808 taler-harness
   1809 -------------
   1810 
   1811 **taler-harness deployment gen-coin-config** is a tool to simplify Taler configuration generation.
   1812 
   1813 
   1814 **taler-harness deployment gen-coin-config**
   1815 [**-min-amount**=**\ ‌\ *VALUE*]
   1816 [**-max-amount**=**\ ‌\ *VALUE*]