taler-developer-manual.rst (70361B)
1 .. 2 This file is part of GNU TALER. 3 4 Copyright (C) 2014-2025 Taler Systems SA 5 6 TALER is free software; you can redistribute it and/or modify it under the 7 terms of the GNU Affero General Public License as published by the Free Software 8 Foundation; either version 3.0, or (at your option) any later version. 9 10 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 11 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 12 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 13 14 You should have received a copy of the GNU Affero General Public License along with 15 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 16 17 @author Christian Grothoff 18 19 General Developer Manual 20 ######################## 21 22 .. note:: 23 24 This manual contains information for developers working on GNU Taler 25 and related components. It is not intended for a general audience. 26 27 28 Project Overview 29 ================ 30 31 GNU Taler consists of a large (and growing) number of components 32 in various Git repositories. The following list gives a first 33 overview: 34 35 * exchange: core payment processing logic with a REST API, plus various 36 helper processes for interaction with banks and cryptographic 37 computations. Also includes the logic for the auditor and an 38 in-memory "bank" API implementation for testing. 39 40 * libeufin: implementation of the "bank" API using the EBICS protocol 41 used by banks in the EU. Allows an exchange to interact with 42 European banks. 43 44 * taler-magnet-bank: implementation of the "bank" API using the Magnet Bank 45 API. Allows an exchange to interact with Magnet Bank. 46 47 * taler-cyclos: implementation of the "bank" API using the Cyclos API. Allows an exchange to interact with a Cyclos network. 48 49 * taler-wise: implementation of the "bank" API using the Wise API. Allows an exchange to interact with Wise. 50 51 * depolymerization: implementation of the "bank" API on top of 52 blockchains, specifically Bitcoin and Ethereum. Allows an exchange 53 to interact with crypto-currencies. 54 55 * merchant: payment processing backend to be run by merchants, 56 offering a REST API. 57 58 * taler-typescript-core: platform-independent implementation of a wallet to be run by 59 normal users. Includes also the WebExtension for various browsers. 60 Furthermore, includes various single-page apps used by other 61 components (especially as libeufin and merchant). Also includes 62 command-line wallet and tools for testing. 63 64 * taler-android: Android Apps including the Android wallet, the 65 Android point-of-sale App and the Android casher app. 66 67 * taler-ios: iOS wallet App. 68 69 * sync: backup service, provides a simple REST API to allow users to 70 make encrypted backups of their wallet state. 71 72 * donau: donation authority to create receipts for donations to charities 73 for tax authorities 74 75 * anastasis: key escrow service, provides a simple REST API to allow 76 users to distribute encryption keys across multiple providers and 77 define authorization policies for key recovery. 78 79 * taler-mdb: integration of Taler with the multi-drop-bus (MDB) API 80 used by vending machines. Allows Taler payments to be integrated 81 with vending machines. 82 83 * gnu-taler-payment-for-woocommerce: payment plugin for the 84 woocommerce (wordpress) E-commerce solution. 85 86 * talerbarr: Dolibarr (ERP) Integration 87 88 * wordpress-turnstile: Wordpress paywall 89 90 * turnstile: Drupal paywall 91 92 * paivana: reverse-proxy based paywall using static pages 93 94 * taler-embedded: embedded hardware wallet 95 96 * taler-xotp: xOTP hardware and firmware for offline payment confirmation 97 98 * pepsi: mail transport agent with support for encryption and 99 Taler payments for spam protection 100 101 * locust: bug tracker with time-tracking and planning features 102 103 * sakana: service availability monitor for exchanges and monitoring infrastructure 104 (the monitor that monitors the monitoring) 105 106 * twister: man-in-the-middle proxy for tests that require fuzzing a 107 REST/JSON protocol. Used for some of our testing. 108 109 * challenger: implementation of an OAuth 2.0 provider that can be used 110 to verify that a user can receive SMS or E-mail at particular addresses. 111 Used as part of KYC processes of the exchange. 112 113 * taler-mailbox: messaging service used to store and forward payment 114 messages to Taler wallets. 115 116 * taldir: directory service used to lookup Taler wallet addresses for 117 sending invoices or payments to other wallets. 118 119 * taler-merchant-demos: various demonstration services operated at 120 'demo.taler.net', including a simple shop and a donation page. 121 122 * kych: legitimization using Swiss eID (Swiyu) 123 124 * beekeeper: onboarding via passport upload and video-identification with lifeness checks 125 126 * robocop: automatic checking of sanction lists 127 128 There are other important repositories without code, including: 129 130 * gana: Hosted on git.gnunet.org, this repository defines various 131 constants used in the GNU Taler project. 132 133 * taler-docs: documentation, including this very document. 134 135 * marketing: various presentations, papers and other resources for 136 outreach. 137 138 * large-media: very large data objects, such as videos. 139 140 * taler-www: the taler.net website. 141 142 Fundamentals 143 ============ 144 145 Versioning 146 ---------- 147 148 A central rule is to never break anything for any dependency. To accomplish 149 this, we use versioning, of the APIs, database schema and the protocol. The 150 database versioning approach is described in the :ref:`Database schema 151 versioning <DatabaseVersioning>` section. Here, we will focus on API and 152 protocol versioning. 153 154 The key issue we need to solve with protocols and APIs (and that does not 155 apply to database versioning) is being able to introduce and remove features 156 without requiring a flag day where all components must update at the same 157 time. For this, we use GNU libtool style versioning with MAJOR:REVISION:AGE 158 and *not* semantic versioning (SEMVER). With GNU libtool style versioning, 159 first the REVISION should be increased on every change to the respective code. 160 Then, each time a feature is introduced or deprecated, the MAJOR and AGE 161 numbers are increased. Whenever an API is actually removed the AGE number is 162 reduced to match the distance since the removed API was deprecated. Thus, if 163 some client implements version X of the protocol (including not using any APIs 164 that have been deprecated), it is compatible for any implementation where 165 MAJOR is larger or equal to X, and MAJOR minus AGE is smaller or equal to X. 166 REVISION is not used for expected compatibility issues and merely serves to 167 uniquely identify each version (in combination with MAJOR). 168 169 To evolve any implementation, it is thus critical to first of all never 170 just break an existing API or endpoint. The only acceptable modifications 171 are to return additional information (being aware of binary compatibility!) 172 or to accept additional optional arguments (again, in a way that does not 173 break existing users). Thus, the most common way to introduce changes will 174 be the addition of new endpoints. Breaking existing endpoints is only ever 175 at best acceptable while in the process of introducing it and if you are 176 absolutely sure that there are zero users in other components. 177 178 When removing endpoints (or fields being returned), you must first deprecate 179 the existing API (incrementing MAJOR and AGE) and then wait for all clients, 180 including all clients in operation (e.g. Android and iOS Apps, e-commerce 181 integrations, etc.) to upgrade to a protocol implementation above the 182 deprecated MAJOR revision. Only then you should remove the endpoint and reduce 183 AGE. 184 185 To document these changes, please try to use ``@since`` annotations in the API 186 specifications to explain the MAJOR revision when a feature became available, 187 but most importantly use ``@deprecated X`` annotations to indicate that an API 188 was deprecated and will be removed once MAJOR minus AGE is above X. When using 189 an API, use the ``/config`` endpoints to check for compatibility and show a 190 warning if the version(s) you support and the version(s) offered by the server 191 are incompatible. 192 193 194 Tagging and Package Versioning 195 ------------------------------ 196 197 Release tags are of the form ``v${major}.${minor}.${patch}``. Release tags *should* be 198 annotated git tags. 199 200 We usually consider Debian packaging files (in ``debian/``) to be part of a release. 201 When only the Debian packaging files need to be changed, there are two options: 202 203 * Make a new patch release (``v${major}.${minor}.${patch+1}``) 204 * Make a Debian release: 205 206 * Debian version now includes a revision: ``${major}.${minor}.${patch}-${debrevision}`` 207 * The tag is Debian-specific: ``debian-${major}.${minor}.${patch}-${debrevision}`` 208 209 All source repos *should* include a ``contrib/bump`` script that automates bumping the\ 210 version in all relevant source and packaging files. 211 In the future, we might add an option to the script to only release a packaging bump. 212 Right now, that process is manual. 213 214 We support tagged and published pre-release versions via tags of the form ``v${major}.${minor}.${patch}-dev.${n}``. 215 The corresponding Debian version must be ``${major}.${minor}.${patch}~dev${n}``. 216 217 Nightly Debian packages built by Buildbot use the highest eligible SemVer 218 tag and the snapshot mapping described in 219 :ref:`ci-debian-package-versions`. The system administration documentation 220 also covers checkout caching and diagnosing version-generation failures. 221 222 Testing Tools 223 ------------- 224 225 For full ``make check`` support, install these programs: 226 227 - `jq <https://github.com/stedolan/jq>`__ 228 - `curl <http://curl.haxx.se>`__ 229 - `faketime <https://github.com/wolfcw/libfaketime>`__ 230 231 The ``make check`` should be able to function without them, but 232 their presence permits some tests to run that would otherwise be skipped. 233 234 Manual Testing Database Reset 235 ----------------------------- 236 237 Sometimes ``make check`` will fail with some kind of database (SQL) 238 error, perhaps with a message like ``OBJECT does not exist`` in the 239 ``test-suite.log`` file, where ``OBJECT`` is the name of a table or function. 240 In that case, it may be necessary to reset the ``talercheck`` database 241 with the commands: 242 243 .. code-block:: console 244 245 $ dropdb talercheck 246 $ createdb talercheck 247 248 This is because, at the moment, there is no support for 249 doing these steps automatically in the ``make check`` flow. 250 251 (If ``make check`` still fails after the reset, file a bug report as usual.) 252 253 Bug Tracking 254 ------------ 255 256 Bug tracking is done with Mantis (https://www.mantisbt.org/). The bug tracker 257 is available at `<https://bugs.taler.net>`_. A registration on the Web site is 258 needed in order to use the bug tracker, only read access is granted without a 259 login. 260 261 We use the following conventions for the bug states: 262 263 * NEW: Incoming bugs are in 'new' so that management (or developers) 264 can easily identify those that need to be checked (report correct? 265 something we want to fix?), prioritized and targeted for releases. 266 "NEW" bugs are never assigned to a developer. 267 268 * FEEDBACK: When blocked on feedback from reporter or other developer. 269 Assigned to other developer (but cannot be assigned to reporter, 270 in this case MAY remain associated with the developer who expects 271 the feedback). If a bug is on feedback, it automatically should be 272 considered to be high-priority to give the feedback (as it is 273 blocking someone else!). 274 275 * ACKNOWLEDGED: The bug has been reviewed, but no decision about 276 what action to take has been made yet. Should not be worked on 277 until management (or a developer) comes up with a plan. 278 "ACKNOWLEDGED" bugs should NOT be assigned to a developer. 279 280 * CONFIRMED: This is a real issue that should be worked on, but 281 is not yet actively worked on. If working on this bug requires 282 other bugs to be fixed first, they should be added as 283 child-bugs (via relationships). Developers are always welcome 284 to self-assign bugs that are "CONFIRMED" if they start to work 285 on a bug. "CONFIRMED" bugs should NOT be assigned to a developer. 286 287 * ASSIGNED: The specific developer the bug is assigned to is 288 **actively** working on the issue. Developers should strive to 289 not have more than 5-10 bugs assigned to them at any time. 290 Only having one assigned to you is totally OK! 291 Developers should aggressively un-assign bugs that they are 292 blocked on, cannot make progress on, or are no longer actively 293 working on (but of course, better *resolve* them before 294 moving on if possible). If the bug remains open, it probably 295 should go back to "CONFIRMED" or "ACKNOWLEDGED". 296 297 * RESOLVED: The bug has been fixed in Git. 298 299 * CLOSED: An official release was made with the fix in it. 300 301 When developers want to keep an eye on certain bugs, they should 302 **monitor** them. Multiple developers can be monitoring a bug, but 303 it can only be assigned to one. Developers should also keep an 304 eye on the roadmap (by release), bug categories they care about, 305 and of course priorities / severities. 306 307 We use **tags** to categorize bugs. Common tags that also imply 308 some urgency include (in alphabetical order): 309 310 * accounting: issues required for accounting (such as taxes by merchants) 311 * compliance: issues related to regulatory compliance 312 * $CUSTOMER: issues requested by a particular customer 313 * performance: performance problems or ideas for improvement 314 * security: security issues (including planned improvements to security) 315 * UX: user experience issues 316 317 These tags **should** be attached to "NEW" bugs if they apply. 318 319 320 Code Repositories 321 ----------------- 322 323 Taler code is versioned with Git. For those users without write access, all the 324 codebases are found at the following URL: 325 326 .. code-block:: none 327 328 git://git.taler.net/<repository> 329 330 A complete list of all the existing repositories is currently found at 331 `<https://git.taler.net/>`_. 332 333 334 Committing code 335 --------------- 336 337 Before you can obtain Git write access, you must sign the copyright 338 agreement. As we collaborate closely with GNUnet, we use their 339 copyright agreement -- with the understanding that your contributions 340 to GNU Taler are included in the assignment. You can find the 341 agreement on the `GNUnet site <https://gnunet.org/en/copyright.html>`_. 342 Please sign and mail it to Christian Grothoff as he currently collects 343 all the documents for GNUnet e.V. 344 345 To obtain Git access, you need to send us your SSH public key. Most core 346 team members have administrative Git access, so simply contact whoever 347 is your primary point of contact so far. You can 348 find instructions on how to generate an SSH key 349 in the `Git book <https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key>`_. 350 If you have been granted write access, you first of all must change the URL of 351 the respective repository to: 352 353 .. code-block:: none 354 355 ssh://git@git.taler.net/<repository> 356 357 For an existing checkout, this can be done by editing the ``.git/config`` file. 358 359 The server is configured to reject all commits that have not been signed with 360 GnuPG. If you do not yet have a GnuPG key, you must create one, as explained 361 in the `GNU Privacy Handbook <https://www.gnupg.org/gph/en/manual/c14.html>`_. 362 You do not need to share the respective public key with us to make commits. 363 However, we recommend that you upload it to key servers, put it on your 364 business card and personally meet with other GNU hackers to have it signed 365 such that others can verify your commits later. 366 367 To sign all commits, you should run 368 369 .. code-block:: console 370 371 $ git config --global commit.gpgsign true 372 373 You can also sign individual commits only by adding the ``-S`` option to the 374 ``git commit`` command. If you accidentally already made commits but forgot 375 to sign them, you can retroactively add signatures using: 376 377 .. code-block:: console 378 379 $ git rebase -S 380 381 382 Whether you commit to a personal branch (recommended: ``dev/$USER/...``), 383 a feature branch or to ``master`` should 384 depend on your level of comfort and the nature of the change. As a general 385 rule, the code in ``master`` must always build and tests should always pass, at 386 least on your own system. However, we all make mistakes and you should expect 387 to receive friendly reminders if your change did not live up to this simple 388 standard. We plan to move to a system where the CI guarantees this invariant 389 in the future. 390 391 In order to keep a linear and clean commits history, we advise to avoid 392 merge commits and instead always rebase your changes before pushing to 393 the ``master`` branch. If you commit and later find out that new commits were 394 pushed, the following command will pull the new commits and rebase yours 395 on top of them. 396 397 .. code-block:: console 398 399 # -S instructs Git to (re)sign your commits 400 $ git pull --rebase -S 401 402 403 404 Observing changes 405 ----------------- 406 407 Every commit to the ``master`` branch of any of our public repositories 408 (and almost all are public) is automatically sent to the 409 gnunet-svn@gnu.org mailinglist. That list is for Git commits only, 410 and must not be used for discussions. It also carries commits from 411 our main dependencies, namely GNUnet and GNU libmicrohttpd. While 412 it can be high volume, the lists is a good way to follow overall 413 development. 414 415 416 Code generator usage policy 417 --------------------------- 418 419 We do neither encourage nor discourage the use of tools for code generation. 420 It is up to the individual developer to decide if a tool is acceptable for 421 a particular task. But of course, we do encourage you to use FLOSS tools 422 and we MUST NOT become dependent on non-free software! That said, if you 423 use tools, you must document their use and in particular satisfy the 424 `NLnet policy on the use of "AI" <https://nlnet.nl/news/2025/20250829-policy-on-use-of-AI.html>`__. 425 426 Specifically, we ask developers to always put generated code into a *separate* 427 Git commit and to include the full prompt in the commit message. Naturally, 428 you may clean up the code generator's output, but then you should do so in 429 separate Git commits (and of course only merge into master/stable after the 430 clean up is complete). But do preserve (not squash!) the commit with the 431 generated code so that it remains documented what the prompts were and which 432 code is generated. This will go a long way to keep code auditors sane! 433 434 435 Communication 436 ------------- 437 438 For public discussions we use the taler@gnu.org mailinglist. All developers 439 should subscribe to the low-volume Taler mailinglist. There are separate 440 low-volume mailinglists for gnunet-developers (@gnu.org) and for libmicrohttpd 441 (@gnu.org). For internal discussions we use https://mattermost.taler.net/ 442 (invitation only, but also archived). 443 444 445 What to put in bootstrap 446 ------------------------ 447 448 Each repository has a ``bootstrap`` script, which contains commands for the 449 developer to run after a repository checkout (i.e., after ``git clone`` or 450 ``git pull``). 451 Typically, this updates and initializes submodules, prepares the tool chain, 452 and runs ``autoreconf``. 453 The last step generates the ``configure`` script, whether for immediate use or 454 for inclusion in the distribution tarball. 455 456 One common submodule is ``contrib/gana``, which pulls from the 457 `GNUnet GANA repository <https://git.gnunet.org/gana.git/>`__. 458 For example, in the 459 `Taler exchange repository <https://git.taler.net/exchange.git>`__, 460 the bootstrap script eventually runs the ``git submodule update --init`` command 461 early on, and later runs script ``./contrib/gana-generate.sh``, which 462 generates files such as ``src/include/taler_signatures.h``. 463 464 Thus, to update that file, you need to: 465 466 - (in GANA repo) Find a suitable (unused) name and number for the Signature 467 Purposes database. 468 469 - Add it to GANA, in ``gnunet-signatures/registry.rec``. 470 (You can check for uniqueness with the ``recfix`` utility.) 471 472 - Commit the change, and push it to the GANA Git repo. 473 474 - (in Taler Repo) Run the ``contrib/gana-latest.sh`` script. 475 476 - Bootstrap, configure, do ``make install``, ``make check``, etc. 477 (Basically, make sure the change does not break anything.) 478 479 - Commit the submodule change, and push it to the Taler exchange Git repo. 480 481 A similar procedure is required for other databases in GANA. 482 See file ``README`` in the various directories for specific instructions. 483 484 485 Debian and Ubuntu Repositories 486 ============================== 487 488 We package our software for Debian and Ubuntu. 489 490 Nightly Repositories 491 -------------------- 492 493 To try the latest, unstable and untested versions of packages, 494 you can add the nightly package sources. 495 496 .. code-block:: shell-session 497 498 # For Debian (trixie) 499 $ curl -sS https://deb.taler.net/apt-nightly/taler-trixie-ci.sources \ 500 | tee /etc/apt/sources.list.d/taler-trixie-nightly.sources 501 502 503 Taler Deployment on gv.taler.net 504 ================================ 505 506 This section describes the GNU Taler deployment on ``gv.taler.net``. ``gv`` 507 is our server at BFH. It hosts the Git repositories, Web sites, CI and other 508 services. Developers can receive an SSH account and e-mail alias for the 509 system, you should contact Javier, Christian or Florian. As with Git, ask 510 your primary team contact for shell access if you think you need it. 511 512 513 DNS 514 --- 515 516 DNS records for taler.net are controlled by the GNU Taler maintainers, 517 specifically Christian and Florian, and our system administrator, Javier. If 518 you need a sub-domain to be added, please contact one of them. 519 520 521 User Acccounts 522 -------------- 523 524 On ``gv.taler.net``, there are three system users that are set up to 525 serve Taler on the Internet: 526 527 - ``head``: serves ``*.head.taler.net`` and gets automatically 528 built by Buildbot every 2 hours from the ``sandcastle-ng.git``. 529 Master key may be reset occasionally 530 531 - ``taler-test``: serves ``*.test.taler.net`` and does *NOT* get 532 automatically built, and runs more recent tags and/or unreleased 533 versions of Taler components. Master key may be reset 534 occasionally. 535 536 - ``demo``: serves ``*.demo.taler.net``. Never automatically built. 537 Master key is retained. 538 539 Demo Upgrade Procedure 540 ====================== 541 542 #. Login as the ``demo`` user on ``gv.taler.net``. 543 #. Pull the latest ``sandcastle-ng.git`` code in checkout at ``$HOME/sandcastle-ng``. 544 #. Run ``systemctl --user restart container-taler-sandcastle-demo.service`` 545 #. Refer to the sandcastle-ng README (https://git.taler.net/sandcastle-ng.git/about/) 546 for more info. 547 548 549 Upgrading the ``demo`` environment should be done with care, and ideally be 550 coordinated on the mailing list before. It is our goal for ``demo`` to always 551 run a "working version" that is compatible with various published wallets. 552 Please use the :doc:`demo upgrade checklist <../checklists/checklist-demo-upgrade>` to make 553 sure everything is working. 554 Nginx is already configured to reach the services as exported by the user unit. 555 556 557 Tagging components 558 ------------------ 559 560 All Taler components must be tagged with git before they are deployed on the 561 ``demo`` environment, using a tag of the following form: 562 563 .. code-block:: none 564 565 demo-YYYY-MM-DD-SS 566 YYYY = year 567 MM = month 568 DD = day 569 SS = serial 570 571 Environments and Builders on taler.net 572 ====================================== 573 574 Buildbot implementation 575 ----------------------- 576 577 GNU Taler uses a buildbot implementation (front end at https://buildbot.taler.net) to manage continuous integration. Buildbot documentation is at https://docs.buildbot.net/. 578 579 Here are some highlights: 580 581 - The WORKER is the config that that lives on a shell account on a localhost (taler.net), where this host has buildbot-worker installed. The WORKER executes the commands that perform all end-functions of buildbot. 582 583 - The WORKER running buildbot-worker receives these commands by authenticating and communicating with the buildbot server using parameters that were specified when the worker was created in that shell account with the ``buildbot-worker`` command. 584 585 - The buildbot server's master.cfg file contains FACTORY declarations which specify the commands that the WORKER will run on localhost. 586 587 - The FACTORY is tied to the WORKER in master.cfg by a BUILDER. 588 589 - The master.cfg also allows for SCHEDULER that defines how and when the BUILDER is executed. 590 591 - Our master.cfg file is checked into git, and then periodically updated on a particular account on taler.net (ask Christian for access if needed). Do not edit this file directly/locally on taler.net, but check changes into Git. 592 593 594 Best Practices: 595 596 - When creating a new WORKER in the ``master.cfg`` file, leave a comment specifying the server and user account that this WORKER is called from. (At this time, taler.net is the only server used by this implementation, but it's still good practice.) 597 598 - Create a worker from a shell account with this command: ``buildbot-worker create-worker <workername> localhost <username> <password>`` 599 600 Then make sure there is a WORKER defined in master.cfg like: ``worker.Worker("<username>", "<password>")`` 601 602 Test builder 603 ------------ 604 605 This builder (``test-builder``) compiles and starts every Taler component. 606 The associated worker is run by the ``taler-test`` Gv user, via the SystemD 607 unit ``buildbot-worker-taler``. The following commands start/stop/restart 608 the worker: 609 610 .. code-block:: 611 612 systemctl --user start buildbot-worker-taler 613 systemctl --user stop buildbot-worker-taler 614 systemctl --user restart buildbot-worker-taler 615 616 .. note:: 617 the mentioned unit file can be found at ``deployment.git/systemd-services/`` 618 619 Wallet builder 620 -------------- 621 622 This builder (``wallet-builder``) compiles every Taler component 623 and runs the wallet integration tests. The associated worker is 624 run by the ``walletbuilder`` Gv user, via the SystemD unit ``buildbot-worker-wallet``. 625 The following commands start/stop/restart the worker: 626 627 .. code-block:: 628 629 systemctl --user start buildbot-worker-wallet 630 systemctl --user stop buildbot-worker-wallet 631 systemctl --user restart buildbot-worker-wallet 632 633 .. note:: 634 the mentioned unit file can be found at ``deployment.git/systemd-services/`` 635 636 Documentation Builder 637 --------------------- 638 639 All the Taler documentation is built by the user ``docbuilder`` that 640 runs a Buildbot worker. The following commands set the ``docbuilder`` up, 641 starting with an empty home directory. 642 643 .. code-block:: console 644 645 # Log-in as the 'docbuilder' user. 646 647 $ cd $HOME 648 $ git clone git://git.taler.net/deployment 649 $ ./deployment/bootstrap-docbuilder 650 651 # If the previous step worked, the setup is 652 # complete and the Buildbot worker can be started. 653 654 $ buildbot-worker start worker/ 655 656 657 Website Builder 658 --------------- 659 660 661 Taler Websites, ``www.taler.net`` and ``stage.taler.net``, are built by the 662 user ``taler-websites`` by the means of a Buildbot worker. The following 663 commands set the ``taler-websites`` up, starting with an empty home directory. 664 665 .. code-block:: console 666 667 # Log-in as the 'taler-websites' user. 668 669 $ cd $HOME 670 $ git clone git://git.taler.net/deployment 671 $ ./deployment/bootstrap-sitesbuilder 672 673 # If the previous step worked, the setup is 674 # complete and the Buildbot worker can be started. 675 676 $ buildbot-worker start worker/ 677 678 679 Code coverage 680 ------------- 681 682 Code coverage tests are run by the ``lcovworker`` user, and are also driven 683 by Buildbot. 684 685 .. code-block:: console 686 687 # Log-in as the 'lcovworker' user. 688 689 $ cd $HOME 690 $ git clone git://git.taler.net/deployment 691 $ ./deployment/bootstrap-taler lcov 692 693 # If the previous step worked, the setup is 694 # complete and the Buildbot worker can be started. 695 696 $ buildbot-worker start worker/ 697 698 The results are then published at ``https://lcov.taler.net/``. 699 700 Producing auditor reports 701 ------------------------- 702 703 Both 'test' and 'demo' setups get their auditor reports compiled 704 by a Buildbot worker. The following steps get the reports compiler 705 prepared. 706 707 .. code-block:: console 708 709 # Log-in as <env>-auditor, with <env> being either 'test' or 'demo' 710 711 $ git clone git://git.taler.net/deployment 712 $ ./deployment/buildbot/bootstrap-scripts/prepare-auditorreporter <env> 713 714 # If the previous steps worked, then it should suffice to start 715 # the worker, with: 716 717 $ buildbot-worker start worker/ 718 719 720 .. _DatabaseVersioning: 721 722 Database schema versioning 723 -------------------------- 724 725 The PostgreSQL databases of the exchange, auditor, and merchant are versioned. 726 See the ``versioning.sql`` file in the respective directory for documentation. 727 728 Every set of changes to the database schema must be stored in a new 729 versioned SQL script. The scripts must have contiguous numbers. After 730 any release (or version being deployed to a production or staging 731 environment), existing scripts MUST be immutable. 732 733 Developers and operators MUST NOT make changes to database schema 734 outside of this versioning. All tables of a GNU Taler component should live in their own schema. 735 736 Runtime stored procedures are loaded from the component's generated 737 ``procedures.sql`` after schema patches. Updating a runtime procedure does 738 not itself advance the schema-patch version; table, index, constraint, and 739 data migrations still require a new numbered script. Component database 740 initialization tools are the supported way to apply both the pending patches 741 and the current procedure definitions. 742 743 744 QA Plans 745 ======== 746 747 .. include:: ../checklists/qa-1.0.rst 748 749 750 Releases 751 ======== 752 753 .. include:: ../checklists/checklist-release.rst 754 755 Release Process 756 --------------- 757 758 This document describes the process for releasing a new version of the 759 various Taler components to the official GNU mirrors. 760 761 The following components are published on the GNU mirrors 762 763 - taler-exchange (exchange.git) 764 - taler-merchant (merchant.git) 765 - sync (sync.git) 766 - taler-mdb (taler-mdb.git) 767 - libeufin (libeufin.git) 768 - challenger (challenger.git) 769 - wallet-core (wallet-core.git) 770 771 Tagging 772 ------- 773 774 Tag releases with an **annotated** commit, like 775 776 .. code-block:: console 777 778 $ git tag -a v0.1.0 -m "Official release v0.1.0" 779 $ git push origin v0.1.0 780 781 Prebuilt artifact policy 782 ------------------------ 783 784 Prebuilt artifacts are stored separately from source, normally on an orphan 785 ``prebuilt`` branch, and are consumed through a Git submodule pinned to an 786 exact commit. Projects producing such artifacts MUST provide repeatable 787 Makefile targets that build the artifact, prepare the prebuilt worktree, and 788 install the result in that worktree. A consumer's ``bootstrap`` script MUST 789 initialize the submodule and SHOULD fail gracefully when an obsolete prebuilt 790 commit is no longer available. 791 792 Artifacts use the layout ``COMPONENT/VERSION/``. ``VERSION`` MUST match both 793 the component version and a source tag. Every revision intended for consumers 794 MUST have a ``prebuilt-SERIAL`` tag; a ``COMPONENT/VERSION`` tag MAY additionally 795 identify the component-specific payload. Consumers MUST pin a tagged commit 796 and SHOULD use sparse checkout. Source, build instructions, and dependency 797 versions must remain sufficient to reproduce the artifact offline. 798 799 800 Database for tests 801 ------------------ 802 803 For tests in the exchange and merchant to run, make sure that a database 804 *talercheck* is accessible by *$USER*. Otherwise tests involving the 805 database logic are skipped. 806 807 .. include:: ../frags/db-stores-sensitive-data.rst 808 809 Exchange, merchant 810 ------------------ 811 812 Set the version in ``configure.ac``. The commit being tagged should be 813 the change of the version. 814 815 Tag the current GANA version that works with the exchange and merchant and 816 checkout that tag of gana.git (instead of master). Otherwise, if there are 817 incompatible changes in GANA (like removed symbols), old builds could break. 818 819 Update the Texinfo documentation using the files from docs.git: 820 821 .. code-block:: console 822 823 # Get the latest documentation repository 824 $ cd $GIT/docs 825 $ git pull 826 $ make texinfo 827 # The *.texi files are now in _build/texinfo 828 # 829 # This checks out the prebuilt branch in the prebuilt directory 830 $ git worktree add prebuilt prebuilt 831 $ cd prebuilt 832 # Copy the pre-built documentation into the prebuilt directory 833 $ cp -r ../_build/texinfo . 834 # Push and commit to branch 835 $ git commit -a -S -m "updating texinfo" 836 $ git status 837 # Verify that all files that should be tracked are tracked, 838 # new files will have to be added to the Makefile.am in 839 # exchange.git as well! 840 $ git push 841 # Remember $REVISION of commit 842 # 843 # Go to exchange 844 $ cd $GIT/exchange/doc/prebuilt 845 # Update submodule to point to latest commit 846 $ git checkout $REVISION 847 848 Finally, the Automake ``Makefile.am`` files may have to be adjusted to 849 include new ``*.texi`` files or images. 850 851 For bootstrap, you will need to install 852 `GNU Recutils <https://www.gnu.org/software/recutils/>`_. 853 854 For the exchange test cases to pass, ``make install`` must be run first. 855 Without it, test cases will fail because plugins can't be located. 856 857 .. code-block:: console 858 859 $ ./bootstrap 860 $ ./configure # add required options for your system 861 $ make dist 862 $ tar -xf taler-$COMPONENT-$VERSION.tar.gz 863 $ cd taler-$COMPONENT-$VERSION 864 $ make install check 865 866 Wallet WebExtension 867 ------------------- 868 869 The version of the wallet is in *manifest.json*. The ``version_name`` 870 should be adjusted, and *version* should be increased independently on 871 every upload to the WebStore. 872 873 .. code-block:: console 874 875 $ ./configure 876 $ make dist 877 878 Upload to GNU mirrors 879 --------------------- 880 881 See https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads 882 883 Directive file: 884 885 .. code-block:: none 886 887 version: 1.2 888 directory: taler 889 filename: taler-exchange-0.1.0.tar.gz 890 symlink: taler-exchange-0.1.0.tar.gz taler-exchange-latest.tar.gz 891 892 Upload the files in **binary mode** to the ftp servers. 893 894 895 Creating Debian packages 896 ------------------------ 897 898 Our general setup is based on 899 https://wiki.debian.org/DebianRepository/SetupWithReprepro 900 901 First, update at least the version of the Debian package in 902 debian/changelog, and then run: 903 904 .. code-block:: bash 905 906 $ dpkg-buildpackage -rfakeroot -b -uc -us 907 908 in the respective source directory (GNUnet, exchange, merchant) to create the 909 ``.deb`` files. Note that they will be created in the parent directory. This 910 can be done on gv.taler.net, or on another (secure) machine. 911 Actual release builds should be done via the Docker images 912 that can be found in ``deployment.git`` under packaging. 913 914 On ``gv``, we use the ``aptbuilder`` user to manage the reprepro repository. 915 916 Next, the ``*.deb`` files should be copied to gv.taler.net, say to 917 ``/home/aptbuilder/incoming``. Then, run 918 919 .. code-block:: bash 920 921 # cd /home/aptbuilder/apt 922 # reprepro includedeb bullseye ~/incoming/*.deb 923 924 to import all Debian files from ``~/incoming/`` into the ``bullseye`` 925 distribution. If Debian packages were build against other distributions, 926 reprepro may need to be first configured for those and the import command 927 updated accordingly. 928 929 Finally, make sure to clean up ``~/incoming/`` (by deleting the 930 now imported ``*.deb`` files). 931 932 933 934 Continuous integration 935 ====================== 936 937 CI is done with Buildbot (https://buildbot.net/), and builds are 938 triggered by the means of Git hooks. The results are published at 939 https://buildbot.taler.net/ . 940 941 In order to avoid downtimes, CI uses a "blue/green" deployment 942 technique. In detail, there are two users building code on the system, 943 the "green" and the "blue" user; and at any given time, one is running 944 Taler services and the other one is either building the code or waiting 945 for that. 946 947 There is also the possibility to trigger builds manually, but this is 948 only reserved to "admin" users. 949 950 Each repository owns its CI definition under ``contrib/ci``. Jobs are 951 directories named ``contrib/ci/jobs/N-NAME`` and MUST contain ``job.sh``; 952 their numeric prefix determines execution order. Optional job configuration 953 is stored as ``config.ini``. Unless every job selects its own container, the 954 repository MUST provide ``contrib/ci/Containerfile``. Projects SHOULD provide 955 at least separate build and test jobs. 956 957 The repository's ``contrib/ci/ci.sh NAME`` entry point runs an individual job 958 locally in the same containerized environment used by Buildbot. CI behavior 959 belongs in the component repository rather than in a central Buildbot-only 960 configuration. Cross-repository pipelines are separate builders because they 961 require several source trees and have different triggering requirements. 962 963 964 Dynamic form metadata 965 ===================== 966 967 Taler Web UIs that consume backend-provided form descriptions use a versioned 968 ``FormMetadata`` envelope. It contains a stable form ``id``, human-readable 969 ``label`` and optional ``description``, numeric ``version``, and a ``config`` 970 object. JSON configurations use one of two layouts: 971 972 * ``single-column`` with an ordered ``fields`` list; or 973 * ``double-column`` with an ordered ``sections`` list, where each section has a 974 title, optional description, and fields. 975 976 Fields are discriminated by their ``type``. Currently supported field classes 977 include amount, boolean, date, duration, one- and multi-select, text, 978 multi-line text, and toggle controls. Consumers MUST reject unknown or 979 malformed structures through the shared codec instead of interpreting them as 980 arbitrary HTML. The version belongs to the form definition and must change 981 when an incompatible metadata interpretation is introduced. 982 983 984 Internationalisation 985 ==================== 986 987 Internationalisation (a.k.a "translation") is handled using text-based 988 localization files named PO (Portable Object) holding pairs of original and 989 translated strings. 990 991 .. include:: dictionary.rst 992 :start-line: 4 993 994 iOS localization uses the checked-in ``Localizable.xcstrings`` catalogs as the 995 application source and checked-in ``XLIFF/*.xliff`` files as the Weblate 996 interchange representation. The XLIFF ``original`` attributes identify the 997 source ``.xcstrings`` catalogs. Translation updates must round-trip through 998 Xcode's XLIFF import/export support, preserve message identifiers and 999 placeholders, and be followed by a full application build. The older proposed 1000 ``pogen`` conversion through PO files is not the deployed iOS workflow. 1001 1002 1003 iOS Apps 1004 ======== 1005 1006 .. _Build-iOS-from-source: 1007 1008 Building Taler Wallet for iOS from source 1009 ----------------------------------------- 1010 1011 The GNU Taler Wallet iOS app is in 1012 `the official Git repository <https://git.taler.net/taler-ios.git>`__. 1013 1014 Compatibility 1015 ^^^^^^^^^^^^^ 1016 1017 The minimum version of iOS supported is 15.0. 1018 This app runs on all iPhone models at least as new as the iPhone 6S. 1019 1020 1021 Building 1022 ^^^^^^^^ 1023 1024 Before building the iOS wallet, you must first checkout the 1025 `quickjs-tart repo <https://git.taler.net/quickjs-tart.git>`__ 1026 and the 1027 `wallet-core repo <https://git.taler.net/wallet-core.git>`__. 1028 1029 Have all 3 local repos (wallet-core, quickjs-tart, and this one) adjacent at 1030 the same level (e.g. in a "GNU_Taler" folder) 1031 Taler.xcworkspace expects the QuickJS framework sub-project to be at 1032 ``../quickjs-tart/QuickJS-rt.xcodeproj``. 1033 1034 Build wallet-core first: 1035 1036 .. code-block:: shell-session 1037 1038 $ cd wallet-core 1039 $ make embedded 1040 $ open packages/taler-wallet-embedded/dist 1041 1042 then drag or move its product "taler-wallet-core-qjs.mjs" 1043 into your quickjs-tart folder right at the top level. 1044 1045 Open Taler.xcworkspace, and set scheme / target to Taler_Wallet. Build&run... 1046 1047 Don't open QuickJS-rt.xcodeproj or TalerWallet.xcodeproj and build anything 1048 there - all needed libraries and frameworks will be built automatically from 1049 Taler.xcworkspace. 1050 1051 1052 Android Apps 1053 ============ 1054 1055 Android App Nightly Builds 1056 -------------------------- 1057 1058 There are currently three Android apps in 1059 `the official Git repository <https://git.taler.net/taler-android.git>`__: 1060 1061 * Wallet 1062 [`CI <https://git.taler.net/taler-android.git/tree/wallet/.gitlab-ci.yml>`__] 1063 * Merchant PoS Terminal 1064 [`CI <https://git.taler.net/taler-android.git/tree/merchant-terminal/.gitlab-ci.yml>`__] 1065 * Cashier 1066 [`CI <https://git.taler.net/taler-android.git/tree/cashier/.gitlab-ci.yml>`__] 1067 1068 Their git repositories are `mirrored at Gitlab <https://gitlab.com/gnu-taler/taler-android>`__ 1069 to utilize their CI 1070 and `F-Droid <https://f-droid.org>`_'s Gitlab integration 1071 to `publish automatic nightly builds <https://f-droid.org/docs/Publishing_Nightly_Builds/>`_ 1072 for each change on the ``master`` branch. 1073 1074 All three apps publish their builds to the same F-Droid nightly repository 1075 (which is stored as a git repository): 1076 https://gitlab.com/gnu-taler/fdroid-repo-nightly 1077 1078 You can download the APK files directly from that repository 1079 or add it to the F-Droid app for automatic updates 1080 by clicking the following link (on the phone that has F-Droid installed). 1081 1082 `GNU Taler Nightly F-Droid Repository <fdroidrepos://gnu-taler.gitlab.io/fdroid-repo-nightly/fdroid/repo?fingerprint=55F8A24F97FAB7B0960016AF393B7E57E7A0B13C2D2D36BAC50E1205923A7843>`_ 1083 1084 .. note:: 1085 Nightly apps can be installed alongside official releases 1086 and thus are meant **only for testing purposes**. 1087 Use at your own risk! 1088 1089 .. _Build-apps-from-source: 1090 1091 Building apps from source 1092 ------------------------- 1093 1094 Note that this guide is different from other guides for building Android apps, 1095 because it does not require you to run non-free software. 1096 It uses the Merchant PoS Terminal as an example, but works as well for the other apps 1097 if you replace ``merchant-terminal`` with ``wallet`` or ``cashier``. 1098 1099 First, ensure that you have the required dependencies installed: 1100 1101 * Java Development Kit 8 or higher (default-jdk-headless) 1102 * git 1103 * unzip 1104 1105 Then you can get the app's source code using git: 1106 1107 .. code-block:: console 1108 1109 # Start by cloning the Android git repository 1110 $ git clone https://git.taler.net/taler-android.git 1111 1112 # Change into the directory of the cloned repository 1113 $ cd taler-android 1114 1115 # Find out which Android SDK version you will need 1116 $ grep -i compileSdkVersion merchant-terminal/build.gradle 1117 1118 The last command will return something like ``compileSdkVersion 29``. 1119 So visit the `Android Rebuilds <http://android-rebuilds.beuc.net/>`_ project 1120 and look for that version of the Android SDK there. 1121 If the SDK version is not yet available as a free rebuild, 1122 you can try to lower the ``compileSdkVersion`` in the app's ``merchant-terminal/build.gradle`` file. 1123 Note that this might break things 1124 or require you to also lower other versions such as ``targetSdkVersion``. 1125 1126 In our example, the version is ``29`` which is available, 1127 so download the "SDK Platform" package of "Android 10.0.0 (API 29)" 1128 and unpack it: 1129 1130 .. code-block:: console 1131 1132 # Change into the directory that contains your downloaded SDK 1133 $ cd $HOME 1134 1135 # Unpack/extract the Android SDK 1136 $ unzip android-sdk_eng.10.0.0_r14_linux-x86.zip 1137 1138 # Tell the build system where to find the SDK 1139 $ export ANDROID_SDK_ROOT="$HOME/android-sdk_eng.10.0.0_r14_linux-x86" 1140 1141 # Change into the directory of the cloned repository 1142 $ cd taler-android 1143 1144 # Build the merchant-terminal app 1145 $ ./gradlew :merchant-terminal:assembleRelease 1146 1147 If you get an error message complaining about build-tools 1148 1149 > Failed to install the following Android SDK packages as some licences have not been accepted. 1150 build-tools;29.0.3 Android SDK Build-Tools 29.0.3 1151 1152 you can try changing the ``buildToolsVersion`` in the app's ``merchant-terminal/build.gradle`` file 1153 to the latest "Android SDK build tools" version supported by the Android Rebuilds project. 1154 1155 After the build finished successfully, 1156 you will find your APK in ``merchant-terminal/build/outputs/apk/release/``. 1157 1158 Update translations 1159 ------------------- 1160 1161 Translations are managed with Taler's weblate instance: 1162 https://weblate.taler.net/projects/gnu-taler/ 1163 1164 To update translations, enter the taler-android git repository 1165 and ensure that the weblate remote exists: 1166 1167 .. code-block:: console 1168 1169 $ git config -l | grep weblate 1170 1171 If it does not yet exist (empty output), you can add it like this: 1172 1173 .. code-block:: console 1174 1175 $ git remote add weblate https://weblate.taler.net/git/gnu-taler/wallet-android/ 1176 1177 Then you can merge in translations commit from the weblate remote: 1178 1179 .. code-block:: console 1180 1181 # ensure you have latest version 1182 $ git fetch weblate 1183 1184 # merge in translation commits 1185 $ git merge weblate/master 1186 1187 Afterwards, build the entire project from source and test the UI 1188 to ensure that no erroneous translations (missing placeholders) are breaking things. 1189 1190 Release process 1191 --------------- 1192 1193 After extensive testing, the code making up a new release should get a signed git tag. 1194 The current tag format is: 1195 1196 * cashier-$VERSION 1197 * pos-$VERSION 1198 * wallet-$VERSION (where $VERSION has a v prefix) 1199 1200 .. code-block:: console 1201 1202 $ git tag -s $APP-$VERSION 1203 1204 F-Droid 1205 ^^^^^^^ 1206 Nightly builds get published automatically (see above) after pushing code to the official repo. 1207 Actual releases get picked up by F-Droid's official repository via git tags. 1208 So ensure that all releases get tagged properly. 1209 1210 Some information for F-Droid official repository debugging: 1211 1212 * Wallet: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.wallet.fdroid.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.wallet.fdroid/lastbuild>`__] 1213 * Cashier: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.cashier.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.cashier/lastbuild>`__] 1214 * PoS: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.merchantpos.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.merchantpos/lastbuild>`__] 1215 1216 Google Play 1217 ^^^^^^^^^^^ 1218 Google Play uploads are managed via `Fastlane <https://docs.fastlane.tools/getting-started/android/setup/>`__. 1219 Before proceeding, ensure that this is properly set up 1220 and that you have access to the Google Play API. 1221 1222 It is important to have access to the signing keys and Google Play access keys 1223 (JSON) and to ensure that the following environment variables are set 1224 correctly and made available to Fastlane: 1225 1226 .. code-block:: bash 1227 1228 TALER_KEYSTORE_PATH= 1229 TALER_KEYSTORE_PASS= 1230 TALER_KEYSTORE_WALLET_ALIAS= 1231 TALER_KEYSTORE_WALLET_PASS= 1232 TALER_KEYSTORE_POS_ALIAS= 1233 TALER_KEYSTORE_POS_PASS= 1234 TALER_KEYSTORE_CASHIER_ALIAS= 1235 TALER_KEYSTORE_CASHIER_PASS= 1236 TALER_JSON_KEY_FILE= 1237 1238 To release an app, enter into its respective folder and run fastlane: 1239 1240 .. code-block:: console 1241 1242 $ bundle exec fastlane 1243 1244 Then select the deploy option. 1245 1246 All uploads are going to the beta track by default. These can be promoted to 1247 production later or immediately after upload if you feel daring. It is also 1248 important to bump the version and build code with every release. 1249 1250 .. _Code-coverage: 1251 1252 Code Coverage 1253 ============= 1254 1255 Code coverage is done with the Gcov / Lcov 1256 (http://ltp.sourceforge.net/coverage/lcov.php) combo, and it is run 1257 nightly (once a day) by a Buildbot worker. The coverage results are 1258 then published at https://lcov.taler.net/ . 1259 1260 1261 Coding Conventions 1262 ================== 1263 1264 GNU Taler is developed primarily in C, Kotlin, Python, Swift and TypeScript. 1265 1266 Components written in C 1267 ----------------------- 1268 1269 These are the general coding style rules for Taler. 1270 1271 * Baseline rules are to follow GNU guidelines, modified or extended 1272 by the GNUnet style: https://docs.gnunet.org/handbook/gnunet.html#Coding-style 1273 1274 Naming conventions 1275 ^^^^^^^^^^^^^^^^^^ 1276 1277 * include files (very similar to GNUnet): 1278 1279 * if installed, must start with "``taler_``" (exception: platform.h), 1280 and MUST live in src/include/ 1281 * if NOT installed, must NOT start with "``taler_``" and 1282 MUST NOT live in src/include/ and 1283 SHOULD NOT be included from outside of their own directory 1284 * end in "_lib" for "simple" libraries 1285 * end in "_plugin" for plugins 1286 * end in "_service" for libraries accessing a service, i.e. the exchange 1287 1288 * binaries: 1289 1290 * taler-exchange-xxx: exchange programs 1291 * taler-merchant-xxx: merchant programs (demos) 1292 * taler-wallet-xxx: wallet programs 1293 * plugins should be libtaler_plugin_xxx_yyy.so: plugin yyy for API xxx 1294 * libtalerxxx: library for API xxx 1295 1296 * logging 1297 1298 * tools use their full name in GNUNET_log_setup 1299 (i.e. 'taler-exchange-offline') and log using plain 'GNUNET_log'. 1300 * pure libraries (without associated service) use 'GNUNET_log_from' 1301 with the component set to their library name (without lib or '.so'), 1302 which should also be their directory name (i.e. 'util') 1303 * plugin libraries (without associated service) use 'GNUNET_log_from' 1304 with the component set to their type and plugin name (without lib or '.so'), 1305 which should also be their directory name (i.e. 'exchangedb-postgres') 1306 * libraries with associated service) use 'GNUNET_log_from' 1307 with the name of the service, which should also be their 1308 directory name (i.e. 'exchange') 1309 * for tools with ``-l LOGFILE``, its absence means write logs to stderr 1310 1311 * configuration 1312 1313 * same rules as for GNUnet 1314 1315 * exported symbols 1316 1317 * must start with TALER_[SUBSYSTEMNAME]_ where SUBSYSTEMNAME 1318 MUST match the subdirectory of src/ in which the symbol is defined 1319 * from libtalerutil start just with ``TALER_``, without subsystemname 1320 * if scope is ONE binary and symbols are not in a shared library, 1321 use binary-specific prefix (such as TMH = taler-exchange-httpd) for 1322 globals, possibly followed by the subsystem (TMH_DB_xxx). 1323 1324 * structs: 1325 1326 * structs that are 'packed' and do not contain pointers and are 1327 thus suitable for hashing or similar operations are distinguished 1328 by adding a "P" at the end of the name. (NEW) Note that this 1329 convention does not hold for the GNUnet-structs (yet). 1330 * structs that are used with a purpose for signatures, additionally 1331 get an "S" at the end of the name. 1332 1333 * private (library-internal) symbols (including structs and macros) 1334 1335 * must not start with ``TALER_`` or any other prefix 1336 1337 * testcases 1338 1339 * must be called "test_module-under-test_case-description.c" 1340 1341 * performance tests 1342 1343 * must be called "perf_module-under-test_case-description.c" 1344 1345 Shell Scripts 1346 ------------- 1347 1348 Shell scripts should be avoided if at all possible. The only permissible uses of shell scripts 1349 in GNU Taler are: 1350 1351 * Trivial invocation of other commands. 1352 * Scripts for compatibility (e.g. ``./configure``) that must run on 1353 as many systems as possible. 1354 1355 When shell scripts are used, they ``MUST`` begin with the following ``set`` command: 1356 1357 .. code-block:: console 1358 1359 # Make the shell fail on undefined variables and 1360 # commands with non-zero exit status. 1361 $ set -eu 1362 1363 Kotlin 1364 ------ 1365 1366 We so far have no specific guidelines, please follow best practices 1367 for the language. 1368 1369 1370 Python 1371 ------ 1372 1373 Supported Python Versions 1374 ^^^^^^^^^^^^^^^^^^^^^^^^^ 1375 1376 Python code should be written and built against version 3.7 of Python. 1377 1378 Style 1379 ^^^^^ 1380 1381 We use `yapf <https://github.com/google/yapf>`_ to reformat the 1382 code to conform to our style instructions. 1383 A reusable yapf style file can be found in ``build-common``, 1384 which is intended to be used as a git submodule. 1385 1386 Python for Scripting 1387 ^^^^^^^^^^^^^^^^^^^^ 1388 1389 When using Python for writing small utilities, the following libraries 1390 are useful: 1391 1392 * ``click`` for argument parsing (should be preferred over argparse) 1393 * ``pathlib`` for path manipulation (part of the standard library) 1394 * ``subprocess`` for "shelling out" to other programs. Prefer ``subprocess.run`` 1395 over the older APIs. 1396 1397 1398 Swift 1399 ----- 1400 1401 Please follow best practices for the language. 1402 1403 1404 TypeScript 1405 ---------- 1406 1407 Please follow best practices for the language. 1408 1409 1410 Testing library 1411 =============== 1412 1413 This chapter is a VERY ABSTRACT description of how testing is 1414 implemented in Taler, and in NO WAY wants to substitute the reading of 1415 the actual source code by the user. 1416 1417 In Taler, a test case is an array of ``struct TALER_TESTING_Command``, 1418 informally referred to as ``CMD``, that is iteratively executed by the 1419 testing interpreter. This latter is transparently initiated by the 1420 testing library. 1421 1422 However, the developer does not have to define CMDs manually, but 1423 rather call the proper constructor provided by the library. For example, 1424 if a CMD is supposed to test feature ``x``, then the library would 1425 provide the ``TALER_TESTING_cmd_x ()`` constructor for it. Obviously, 1426 each constructor has its own particular arguments that make sense to 1427 test ``x``, and all constructors are thoroughly commented within the 1428 source code. 1429 1430 Internally, each CMD has two methods: ``run ()`` and ``cleanup ()``. The 1431 former contains the main logic to test feature ``x``, whereas the latter 1432 cleans the memory up after execution. 1433 1434 In a test life, each CMD needs some internal state, made by values it 1435 keeps in memory. Often, the test has to *share* those values with other 1436 CMDs: for example, CMD1 may create some key material and CMD2 needs this 1437 key material to encrypt data. 1438 1439 The offering of internal values from CMD1 to CMD2 is made by *traits*. A 1440 trait is a ``struct TALER_TESTING_Trait``, and each CMD contains an array 1441 of traits, that it offers via the public trait interface to other 1442 commands. The definition and filling of such array happens transparently 1443 to the test developer. 1444 1445 For example, the following example shows how CMD2 takes an amount object 1446 offered by CMD1 via the trait interface. 1447 1448 Note: the main interpreter and the most part of CMDs and traits are 1449 hosted inside the exchange codebase, but nothing prevents the developer 1450 from implementing new CMDs and traits within other codebases. 1451 1452 .. code-block:: c 1453 1454 /* Without loss of generality, let's consider the 1455 * following logic to exist inside the run() method of CMD1 */ 1456 ... 1457 1458 struct TALER_Amount *a; 1459 /** 1460 * the second argument (0) points to the first amount object offered, 1461 * in case multiple are available. 1462 */ 1463 if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a)) 1464 return GNUNET_SYSERR; 1465 ... 1466 1467 use(a); /* 'a' points straight into the internal state of CMD2 */ 1468 1469 In the Taler realm, there is also the possibility to alter the behaviour 1470 of supposedly well-behaved components. This is needed when, for example, 1471 we want the exchange to return some corrupted signature in order to 1472 check if the merchant backend detects it. 1473 1474 This alteration is accomplished by another service called *twister*. The 1475 twister acts as a proxy between service A and B, and can be programmed 1476 to tamper with the data exchanged by A and B. 1477 1478 Please refer to the Twister codebase (under the ``test`` directory) in 1479 order to see how to configure it. 1480 1481 1482 User-Facing Terminology 1483 ======================= 1484 1485 This section contains terminology that should be used and that should not be 1486 used in the user interface and help materials. 1487 1488 Terms to Avoid 1489 -------------- 1490 1491 Refreshing 1492 Refreshing is the internal technical terminology for the protocol to 1493 give change for partially spent coins 1494 1495 **Use instead**: "Obtaining change" 1496 1497 Charge 1498 Charge has two opposite meanings (charge to a credit card vs. charge a battery). 1499 This can confuse users. 1500 1501 **Use instead**: "Obtain", "Credit", "Debit", "Withdraw", "Top up" 1502 1503 Coin 1504 Coins are an internal construct, the user should never be aware that their balance 1505 is represented by coins of different denominations. 1506 1507 **Use instead**: "(Digital) Cash" or "(Wallet) Balance" 1508 1509 Consumer 1510 Has bad connotation of consumption. 1511 1512 **Use instead**: Customer or user. 1513 1514 Proposal 1515 The term used to describe the process of the merchant facilitating the download 1516 of the signed contract terms for an order. 1517 1518 **Avoid**. Generally events that relate to proposal downloads 1519 should not be shown to normal users, only developers. Instead, use 1520 "communication with merchant failed" if a proposed order can't be downloaded. 1521 1522 Anonymous E-Cash 1523 Should be generally avoided, since Taler is only anonymous for 1524 the customer. Also some people are scared of anonymity (which as 1525 a term is also way too absolute, as anonymity is hardly ever perfect). 1526 1527 **Use instead**: "Privacy-preserving", "Privacy-friendly" 1528 1529 Payment Replay 1530 The process of proving to the merchant that the customer is entitled 1531 to view a digital product again, as they already paid for it. 1532 1533 **Use instead**: In the event history, "re-activated digital content purchase" 1534 could be used. (FIXME: this is still not nice.) 1535 1536 Session ID 1537 See Payment Replay. 1538 1539 Order 1540 Too ambiguous in the wallet. 1541 1542 **Use instead**: Purchase 1543 1544 Fulfillment URL 1545 URL that serves the digital content that the user purchased 1546 with their payment. Can also be something like a donation receipt. 1547 1548 Donau 1549 Developer-internal name for the tax authority component. 1550 1551 **Use instead**: Tax authority 1552 1553 Terms to Use 1554 ------------ 1555 1556 Auditor 1557 Regulatory entity that certifies exchanges and oversees their operation. 1558 1559 Exchange Operator 1560 The entity/service that gives out digital cash in exchange for some 1561 other means of payment. 1562 1563 In some contexts, using "Issuer" could also be appropriate. 1564 When showing a balance breakdown, 1565 we can say "100 Eur (issued by exchange.euro.taler.net)". 1566 Sometimes we may also use the more generic term "Payment Service Provider" 1567 when the concept of an "Exchange" is still unclear to the reader. 1568 1569 Refund 1570 A refund is given by a merchant to the customer (rather the customer's wallet) 1571 and "undoes" a previous payment operation. 1572 1573 Payment 1574 The act of sending digital cash to a merchant to pay for an order. 1575 1576 Purchase 1577 Used to refer to the "result" of a payment, as in "view purchase". 1578 Use sparingly, as the word doesn't fit for all payments, such as donations. 1579 1580 Contract Terms 1581 Partially machine-readable representation of the merchant's obligation after the 1582 customer makes a payment. 1583 1584 Merchant 1585 Party that receives a payment. 1586 1587 Wallet 1588 Also "Taler Wallet". Software component that manages the user's digital cash 1589 and payments. 1590 1591 1592 Developer Glossary 1593 ================== 1594 1595 This glossary is meant for developers. It contains some terms that we usually do not 1596 use when talking to end users or even system administrators. 1597 1598 .. glossary:: 1599 :sorted: 1600 1601 absolute time 1602 method of keeping time in :term:`GNUnet` where the time is represented 1603 as the number of microseconds since 1.1.1970 (UNIX epoch). Called 1604 absolute time in contrast to :term:`relative time`. 1605 1606 aggregate 1607 the :term:`exchange` combines multiple payments received by the 1608 same :term:`merchant` into one larger :term:`wire transfer` to 1609 the respective merchant's :term:`bank` account 1610 1611 auditor 1612 trusted third party that verifies that the :term:`exchange` is operating correctly 1613 1614 bank 1615 traditional financial service provider who offers 1616 :term:`wire transfers <wire transfer>` between accounts 1617 1618 buyer 1619 individual in control of a Taler :term:`wallet`, usually using it to 1620 :term:`spend` the :term:`coins <coin>` on :term:`contracts <contract>` (see also :term:`customer`). 1621 1622 close 1623 operation an :term:`exchange` performs on a :term:`reserve` that has not been 1624 :term:`emptied <empty>` by :term:`withdraw` operations. When closing a reserve, the 1625 exchange wires the remaining funds back to the customer, minus a :term:`fee` 1626 for closing 1627 1628 customer 1629 individual that directs the buyer (perhaps the same individual) to make a purchase 1630 1631 coin 1632 coins are individual tokens representing a certain amount of value, also known as the :term:`denomination` of the coin 1633 1634 refresh commitment 1635 data that the wallet commits to during the :term:`melt` stage of the 1636 :term:`refresh` protocol where it 1637 has to prove to the :term:`exchange` that it is deriving the :term:`fresh` 1638 coins as specified by the Taler protocol. The commitment is verified 1639 probabilistically (see: :term:`kappa`) during the :term:`reveal` stage. 1640 1641 contract 1642 formal agreement between :term:`merchant` and :term:`customer` specifying the 1643 :term:`contract terms` and signed by the merchant and the :term:`coins <coin>` of the 1644 customer 1645 1646 contract terms 1647 the individual clauses specifying what the buyer is purchasing from the 1648 :term:`merchant` 1649 1650 denomination 1651 unit of currency, specifies both the currency and the face value of a :term:`coin`, 1652 as well as associated fees and validity periods 1653 1654 denomination key 1655 (RSA) key used by the exchange to certify that a given :term:`coin` is valid and of a 1656 particular :term:`denomination` 1657 1658 deposit 1659 operation by which a merchant passes coins to an exchange, expecting the 1660 exchange to credit his bank account in the future using an 1661 :term:`aggregate` :term:`wire transfer` 1662 1663 drain 1664 process by which an exchange operator takes the profits 1665 (from :term:`fees <fee>`) out of the escrow account and moves them into 1666 their regular business account 1667 1668 dirty 1669 a :term:`coin` is dirty if its public key may be known to an entity other than 1670 the customer, thereby creating the danger of some entity being able to 1671 link multiple transactions of coin's owner if the coin is not refreshed 1672 1673 empty 1674 a :term:`reserve` is being emptied when a :term:`wallet` is using the 1675 reserve's private key to :term:`withdraw` coins from it. This reduces 1676 the balance of the reserve. Once the balance reaches zero, we say that 1677 the reserve has been (fully) emptied. Reserves that are not emptied 1678 (which is the normal process) are :term:`closed <close>` by the exchange. 1679 1680 exchange 1681 Taler's payment service operator. Issues electronic coins during 1682 withdrawal and redeems them when they are deposited by merchants 1683 1684 expired 1685 Various operations come with time limits. In particular, denomination keys 1686 come with strict time limits for the various operations involving the 1687 coin issued under the denomination. The most important limit is the 1688 deposit expiration, which specifies until when wallets are allowed to 1689 use the coin in deposit or refreshing operations. There is also a "legal" 1690 expiration, which specifies how long the exchange keeps records beyond the 1691 deposit expiration time. This latter expiration matters for legal disputes 1692 in courts and also creates an upper limit for refreshing operations on 1693 special zombie coin 1694 1695 GNUnet 1696 Codebase of various libraries for a better Internet, some of which 1697 GNU Taler depends upon. 1698 1699 fakebank 1700 implementation of the :term:`bank` API in memory to be used only for test 1701 cases. 1702 1703 fee 1704 an :term:`exchange` charges various fees for its service. The different 1705 fees are specified in the protocol. There are fees per coin for 1706 :term:`withdrawing <withdraw>`, :term:`depositing <deposit>`, :term:`melting <melt>`, and 1707 :term:`refunding <refund>`. Furthermore, there are fees per wire transfer 1708 when a :term:`reserve` is :term:`closed <close>` 1709 and for :term:`aggregate` :term:`wire transfers <wire transfer>` 1710 to the :term:`merchant`. 1711 1712 fresh 1713 a :term:`coin` is fresh if its public key is only known to the customer 1714 1715 JSON 1716 JavaScript Object Notation (JSON) is a 1717 serialization format derived from the JavaScript language which is 1718 commonly used in the Taler protocol as the payload of HTTP requests 1719 and responses. 1720 1721 kappa 1722 security parameter used in the :term:`refresh` protocol. Defined to be 3. 1723 The probability of successfully evading the income transparency with the 1724 refresh protocol is 1:kappa. 1725 1726 libeufin 1727 Kotlin component that implements a regional currency bank and an 1728 adapter to communicate via EBICS with European core banking systems. 1729 1730 link 1731 specific step in the :term:`refresh` protocol that an exchange must offer 1732 to prevent abuse of the :term:`refresh` mechanism. The link step is 1733 not needed in normal operation, it just must be offered. 1734 1735 master key 1736 offline key used by the exchange to certify denomination keys and 1737 message signing keys 1738 1739 melt 1740 step of the :term:`refresh` protocol where a :term:`dirty` :term:`coin` 1741 is invalidated to be reborn :term:`fresh` in a subsequent 1742 :term:`reveal` step. 1743 1744 merchant 1745 party receiving payments (usually in return for goods or services) 1746 1747 message signing key 1748 key used by the exchange to sign online messages, other than coins 1749 1750 order 1751 offer made by the merchant to a wallet; pre-cursor to 1752 a contract where the wallet is not yet fixed. Turns 1753 into a :term:`contract` when a wallet claims the order. 1754 1755 owner 1756 a coin is owned by the entity that knows the private key of the coin 1757 1758 relative time 1759 method of keeping time in :term:`GNUnet` where the time is represented 1760 as a relative number of microseconds. Thus, a relative time specifies 1761 an offset or a duration, but not a date. Called relative time in 1762 contrast to :term:`absolute time`. 1763 1764 recoup 1765 Operation by which an exchange returns the value of coins affected 1766 by a :term:`revocation <revoke>` to their :term:`owner`, either by allowing the owner to 1767 withdraw new coins or wiring funds back to the bank account of the :term:`owner`. 1768 1769 planchet 1770 precursor data for a :term:`coin`. A planchet includes the coin's internal 1771 secrets (coin private key, blinding factor), but lacks the RSA signature 1772 of the :term:`exchange`. When :term:`withdrawing <withdraw>`, a :term:`wallet` 1773 creates and persists a planchet before asking the exchange to sign it to 1774 get the coin. 1775 1776 purchase 1777 Refers to the overall process of negotiating a :term:`contract` and then 1778 making a payment with :term:`coins <coin>` to a :term:`merchant`. 1779 1780 privacy policy 1781 Statement of an operator how they will protect the privacy of users. 1782 1783 proof 1784 Message that cryptographically demonstrates that a particular claim is correct. 1785 1786 proposal 1787 a list of :term:`contract terms` that has been completed and signed by the 1788 merchant backend. 1789 1790 refresh 1791 operation by which a :term:`dirty` :term:`coin` is converted into one or more 1792 :term:`fresh` coins. Involves :term:`melting <melt>` the :term:`dirty` coins and 1793 then :term:`revealing <reveal>` so-called :term:`transfer keys <transfer key>`. 1794 1795 refund 1796 operation by which a merchant steps back from the right to funds that he 1797 obtained from a :term:`deposit` operation, giving the right to the funds back 1798 to the customer 1799 1800 refund transaction id 1801 unique number by which a merchant identifies a :term:`refund`. Needed 1802 as refunds can be partial and thus there could be multiple refunds for 1803 the same :term:`purchase`. 1804 1805 reserve 1806 accounting mechanism used by the exchange to track customer funds 1807 from incoming :term:`wire transfers <wire transfer>`. A reserve is created whenever 1808 a customer wires money to the exchange using a well-formed public key 1809 in the subject. The exchange then allows the customer's :term:`wallet` 1810 to :term:`withdraw` up to the amount received in :term:`fresh` 1811 :term:`coins <coin>` from the reserve, thereby emptying the reserve. If a 1812 reserve is not emptied, the exchange will eventually :term:`close` it. 1813 1814 Other definition: Funds set aside for future use; either the balance of a customer at the 1815 exchange ready for withdrawal, or the funds kept in the exchange;s bank 1816 account to cover obligations from coins in circulation. 1817 1818 reveal 1819 step in the :term:`refresh` protocol where some of the transfer private 1820 keys are revealed to prove honest behavior on the part of the wallet. 1821 In the reveal step, the exchange returns the signed :term:`fresh` coins. 1822 1823 revoke 1824 exceptional operation by which an exchange withdraws a denomination from 1825 circulation, either because the signing key was compromised or because 1826 the exchange is going out of operation; unspent coins of a revoked 1827 denomination are subjected to recoup. 1828 1829 sharing 1830 users can share ownership of a :term:`coin` by sharing access to the coin's 1831 private key, thereby allowing all co-owners to spend the coin at any 1832 time. 1833 1834 spend 1835 operation by which a customer gives a merchant the right to deposit 1836 coins in return for merchandise 1837 1838 transfer key 1839 special cryptographic key used in the :term:`refresh` protocol, some of which 1840 are revealed during the :term:`reveal` step. Note that transfer keys have, 1841 despite the name, no relationship to :term:`wire transfers <wire transfer>`. They merely 1842 help to transfer the value from a :term:`dirty` coin to a :term:`fresh` coin 1843 1844 terms 1845 the general terms of service of an operator, possibly including 1846 the :term:`privacy policy`. Not to be confused with the 1847 :term:`contract terms` which are about the specific purchase. 1848 1849 transaction 1850 method by which ownership is exclusively transferred from one entity 1851 1852 user 1853 any individual using the Taler payment system 1854 (see :term:`customer`, :term:`buyer`, :term:`merchant`). 1855 1856 version 1857 Taler uses various forms of versioning. There is a database 1858 schema version (stored itself in the database, see \*-0000.sql) describing 1859 the state of the table structure in the database of an :term:`exchange`, 1860 :term:`auditor` or :term:`merchant`. There is a protocol 1861 version (CURRENT:REVISION:AGE, see GNU libtool) which specifies 1862 the network protocol spoken by an :term:`exchange` or :term:`merchant` 1863 including backwards-compatibility. And finally there is the software 1864 release version (MAJOR.MINOR.PATCH, see https://semver.org/) of 1865 the respective code base. 1866 1867 wallet 1868 software running on a customer's computer; withdraws, stores and 1869 spends coins 1870 1871 WebExtension 1872 Cross-browser API used to implement the GNU Taler wallet browser extension. 1873 1874 wire gateway 1875 API used by the exchange to talk with some real-time gross settlement system 1876 (core banking system, blockchain) to notice inbound credits wire transfers 1877 (during withdraw) and to trigger outbound debit wire transfers (primarily 1878 for deposits). 1879 1880 wire transfer 1881 a wire transfer is a method of sending funds between :term:`bank` accounts 1882 1883 wire transfer identifier 1884 Subject of a wire transfer from the exchange to a merchant; 1885 set by the aggregator to a random nonce which uniquely 1886 identifies the transfer. 1887 1888 withdraw 1889 operation by which a :term:`wallet` can convert funds from a :term:`reserve` to 1890 fresh coins 1891 1892 zombie 1893 :term:`coin` where the respective :term:`denomination key` is past its 1894 :term:`deposit` :term:`expiration <expired>` time, but which is still (again) valid 1895 for an operation because it was :term:`melted <melt>` while it was still 1896 valid, and then later again credited during a :term:`recoup` process 1897 1898 1899 1900 Developer Tools 1901 =============== 1902 1903 This section describes various internal programs to make life easier for the 1904 developer. 1905 1906 1907 taler-harness 1908 ------------- 1909 1910 **taler-harness deployment gen-coin-config** is a tool to simplify Taler configuration generation. 1911 1912 1913 **taler-harness deployment gen-coin-config** 1914 [**-min-amount**=**\ \ *VALUE*] 1915 [**-max-amount**=**\ \ *VALUE*]