taler-developer-manual.rst (69469B)
1 .. 2 This file is part of GNU TALER. 3 4 Copyright (C) 2014-2025 Taler Systems SA 5 6 TALER is free software; you can redistribute it and/or modify it under the 7 terms of the GNU Affero General Public License as published by the Free Software 8 Foundation; either version 3.0, or (at your option) any later version. 9 10 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 11 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 12 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 13 14 You should have received a copy of the GNU Affero General Public License along with 15 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 16 17 @author Christian Grothoff 18 19 General Developer Manual 20 ######################## 21 22 .. note:: 23 24 This manual contains information for developers working on GNU Taler 25 and related components. It is not intended for a general audience. 26 27 28 Project Overview 29 ================ 30 31 GNU Taler consists of a large (and growing) number of components 32 in various Git repositories. The following list gives a first 33 overview: 34 35 * exchange: core payment processing logic with a REST API, plus various 36 helper processes for interaction with banks and cryptographic 37 computations. Also includes the logic for the auditor and an 38 in-memory "bank" API implementation for testing. 39 40 * libeufin: implementation of the "bank" API using the EBICS protocol 41 used by banks in the EU. Allows an exchange to interact with 42 European banks. 43 44 * taler-magnet-bank: implementation of the "bank" API using the Magnet Bank 45 API. Allows an exchange to interact with Magnet Bank. 46 47 * taler-cyclos: implementation of the "bank" API using the Cyclos API. Allows an exchange to interact with a Cyclos network. 48 49 * taler-wise: implementation of the "bank" API using the Wise API. Allows an exchange to interact with Wise. 50 51 * depolymerization: implementation of the "bank" API on top of 52 blockchains, specifically Bitcoin and Ethereum. Allows an exchange 53 to interact with crypto-currencies. 54 55 * merchant: payment processing backend to be run by merchants, 56 offering a REST API. 57 58 * wallet-core: platform-independent implementation of a wallet to be run by 59 normal users. Includes also the WebExtension for various browsers. 60 Furthermore, includes various single-page apps used by other 61 components (especially as libeufin and merchant). Also includes 62 command-line wallet and tools for testing. 63 64 * taler-android: Android Apps including the Android wallet, the 65 Android point-of-sale App and the Android casher app. 66 67 * taler-ios: iOS wallet App. 68 69 * sync: backup service, provides a simple REST API to allow users to 70 make encrypted backups of their wallet state. 71 72 * anastasis: key escrow service, provides a simple REST API to allow 73 users to distribute encryption keys across multiple providers and 74 define authorization policies for key recovery. 75 76 * taler-mdb: integration of Taler with the multi-drop-bus (MDB) API 77 used by vending machines. Allows Taler payments to be integrated 78 with vending machines. 79 80 * gnu-taler-payment-for-woocommerce: payment plugin for the 81 woocommerce (wordpress) E-commerce solution. 82 83 * twister: man-in-the-middle proxy for tests that require fuzzing a 84 REST/JSON protocol. Used for some of our testing. 85 86 * challenger: implementation of an OAuth 2.0 provider that can be used 87 to verify that a user can receive SMS or E-mail at particular addresses. 88 Used as part of KYC processes of the exchange. 89 90 * taler-mailbox: messaging service used to store and forward payment 91 messages to Taler wallets. 92 93 * taldir: directory service used to lookup Taler wallet addresses for 94 sending invoices or payments to other wallets. 95 96 * taler-merchant-demos: various demonstration services operated at 97 'demo.taler.net', including a simple shop and a donation page. 98 99 There are other important repositories without code, including: 100 101 * gana: Hosted on git.gnunet.org, this repository defines various 102 constants used in the GNU Taler project. 103 104 * docs: documentation, including this very document. 105 106 * marketing: various presentations, papers and other resources for 107 outreach. 108 109 * large-media: very large data objects, such as videos. 110 111 * www: the taler.net website. 112 113 Fundamentals 114 ============ 115 116 Versioning 117 ---------- 118 119 A central rule is to never break anything for any dependency. To accomplish 120 this, we use versioning, of the APIs, database schema and the protocol. The 121 database versioning approach is described in the :ref:`Database schema 122 versioning <DatabaseVersioning>` section. Here, we will focus on API and 123 protocol versioning. 124 125 The key issue we need to solve with protocols and APIs (and that does not 126 apply to database versioning) is being able to introduce and remove features 127 without requiring a flag day where all components must update at the same 128 time. For this, we use GNU libtool style versioning with MAJOR:REVISION:AGE 129 and *not* semantic versioning (SEMVER). With GNU libtool style versioning, 130 first the REVISION should be increased on every change to the respective code. 131 Then, each time a feature is introduced or deprecated, the MAJOR and AGE 132 numbers are increased. Whenever an API is actually removed the AGE number is 133 reduced to match the distance since the removed API was deprecated. Thus, if 134 some client implements version X of the protocol (including not using any APIs 135 that have been deprecated), it is compatible for any implementation where 136 MAJOR is larger or equal to X, and MAJOR minus AGE is smaller or equal to X. 137 REVISION is not used for expected compatibility issues and merely serves to 138 uniquely identify each version (in combination with MAJOR). 139 140 To evolve any implementation, it is thus critical to first of all never 141 just break an existing API or endpoint. The only acceptable modifications 142 are to return additional information (being aware of binary compatibility!) 143 or to accept additional optional arguments (again, in a way that does not 144 break existing users). Thus, the most common way to introduce changes will 145 be the addition of new endpoints. Breaking existing endpoints is only ever 146 at best acceptable while in the process of introducing it and if you are 147 absolutely sure that there are zero users in other components. 148 149 When removing endpoints (or fields being returned), you must first deprecate 150 the existing API (incrementing MAJOR and AGE) and then wait for all clients, 151 including all clients in operation (e.g. Android and iOS Apps, e-commerce 152 integrations, etc.) to upgrade to a protocol implementation above the 153 deprecated MAJOR revision. Only then you should remove the endpoint and reduce 154 AGE. 155 156 To document these changes, please try to use ``@since`` annotations in the API 157 specifications to explain the MAJOR revision when a feature became available, 158 but most importantly use ``@deprecated X`` annotations to indicate that an API 159 was deprecated and will be removed once MAJOR minus AGE is above X. When using 160 an API, use the ``/config`` endpoints to check for compatibility and show a 161 warning if the version(s) you support and the version(s) offered by the server 162 are incompatible. 163 164 165 Tagging and Package Versioning 166 ------------------------------ 167 168 Release tags are of the form ``v${major}.${minor}.${patch}``. Release tags *should* be 169 annotated git tags. 170 171 We usually consider Debian packaging files (in ``debian/``) to be part of a release. 172 When only the Debian packaging files need to be changed, there are two options: 173 174 * Make a new patch release (``v${major}.${minor}.${patch+1}``) 175 * Make a Debian release: 176 177 * Debian version now includes a revision: ``${major}.${minor}.${patch}-${debrevision}`` 178 * The tag is Debian-specific: ``debian-${major}.${minor}.${patch}-${debrevision}`` 179 180 All source repos *should* include a ``contrib/bump`` script that automates bumping the\ 181 version in all relevant source and packaging files. 182 In the future, we might add an option to the script to only release a packaging bump. 183 Right now, that process is manual. 184 185 We support tagged and published pre-release versions via tags of the form ``v${major}.${minor}.${patch}-dev.${n}``. 186 The corresponding Debian version must be ``${major}.${minor}.${patch}~dev${n}``. 187 188 Nightly Debian packages built by Buildbot use the highest eligible SemVer 189 tag and the snapshot mapping described in 190 :ref:`ci-debian-package-versions`. The system administration documentation 191 also covers checkout caching and diagnosing version-generation failures. 192 193 Testing Tools 194 ------------- 195 196 For full ``make check`` support, install these programs: 197 198 - `jq <https://github.com/stedolan/jq>`__ 199 - `curl <http://curl.haxx.se>`__ 200 - `faketime <https://github.com/wolfcw/libfaketime>`__ 201 202 The ``make check`` should be able to function without them, but 203 their presence permits some tests to run that would otherwise be skipped. 204 205 Manual Testing Database Reset 206 ----------------------------- 207 208 Sometimes ``make check`` will fail with some kind of database (SQL) 209 error, perhaps with a message like ``OBJECT does not exist`` in the 210 ``test-suite.log`` file, where ``OBJECT`` is the name of a table or function. 211 In that case, it may be necessary to reset the ``talercheck`` database 212 with the commands: 213 214 .. code-block:: console 215 216 $ dropdb talercheck 217 $ createdb talercheck 218 219 This is because, at the moment, there is no support for 220 doing these steps automatically in the ``make check`` flow. 221 222 (If ``make check`` still fails after the reset, file a bug report as usual.) 223 224 Bug Tracking 225 ------------ 226 227 Bug tracking is done with Mantis (https://www.mantisbt.org/). The bug tracker 228 is available at `<https://bugs.taler.net>`_. A registration on the Web site is 229 needed in order to use the bug tracker, only read access is granted without a 230 login. 231 232 We use the following conventions for the bug states: 233 234 * NEW: Incoming bugs are in 'new' so that management (or developers) 235 can easily identify those that need to be checked (report correct? 236 something we want to fix?), prioritized and targeted for releases. 237 "NEW" bugs are never assigned to a developer. 238 239 * FEEDBACK: When blocked on feedback from reporter or other developer. 240 Assigned to other developer (but cannot be assigned to reporter, 241 in this case MAY remain associated with the developer who expects 242 the feedback). If a bug is on feedback, it automatically should be 243 considered to be high-priority to give the feedback (as it is 244 blocking someone else!). 245 246 * ACKNOWLEDGED: The bug has been reviewed, but no decision about 247 what action to take has been made yet. Should not be worked on 248 until management (or a developer) comes up with a plan. 249 "ACKNOWLEDGED" bugs should NOT be assigned to a developer. 250 251 * CONFIRMED: This is a real issue that should be worked on, but 252 is not yet actively worked on. If working on this bug requires 253 other bugs to be fixed first, they should be added as 254 child-bugs (via relationships). Developers are always welcome 255 to self-assign bugs that are "CONFIRMED" if they start to work 256 on a bug. "CONFIRMED" bugs should NOT be assigned to a developer. 257 258 * ASSIGNED: The specific developer the bug is assigned to is 259 **actively** working on the issue. Developers should strive to 260 not have more than 5-10 bugs assigned to them at any time. 261 Only having one assigned to you is totally OK! 262 Developers should aggressively un-assign bugs that they are 263 blocked on, cannot make progress on, or are no longer actively 264 working on (but of course, better *resolve* them before 265 moving on if possible). If the bug remains open, it probably 266 should go back to "CONFIRMED" or "ACKNOWLEDGED". 267 268 * RESOLVED: The bug has been fixed in Git. 269 270 * CLOSED: An official release was made with the fix in it. 271 272 When developers want to keep an eye on certain bugs, they should 273 **monitor** them. Multiple developers can be monitoring a bug, but 274 it can only be assigned to one. Developers should also keep an 275 eye on the roadmap (by release), bug categories they care about, 276 and of course priorities / severities. 277 278 We use **tags** to categorize bugs. Common tags that also imply 279 some urgency include (in alphabetical order): 280 281 * accounting: issues required for accounting (such as taxes by merchants) 282 * compliance: issues related to regulatory compliance 283 * $CUSTOMER: issues requested by a particular customer 284 * performance: performance problems or ideas for improvement 285 * security: security issues (including planned improvements to security) 286 * UX: user experience issues 287 288 These tags **should** be attached to "NEW" bugs if they apply. 289 290 291 Code Repositories 292 ----------------- 293 294 Taler code is versioned with Git. For those users without write access, all the 295 codebases are found at the following URL: 296 297 .. code-block:: none 298 299 git://git.taler.net/<repository> 300 301 A complete list of all the existing repositories is currently found at 302 `<https://git.taler.net/>`_. 303 304 305 Committing code 306 --------------- 307 308 Before you can obtain Git write access, you must sign the copyright 309 agreement. As we collaborate closely with GNUnet, we use their 310 copyright agreement -- with the understanding that your contributions 311 to GNU Taler are included in the assignment. You can find the 312 agreement on the `GNUnet site <https://gnunet.org/en/copyright.html>`_. 313 Please sign and mail it to Christian Grothoff as he currently collects 314 all the documents for GNUnet e.V. 315 316 To obtain Git access, you need to send us your SSH public key. Most core 317 team members have administrative Git access, so simply contact whoever 318 is your primary point of contact so far. You can 319 find instructions on how to generate an SSH key 320 in the `Git book <https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key>`_. 321 If you have been granted write access, you first of all must change the URL of 322 the respective repository to: 323 324 .. code-block:: none 325 326 ssh://git@git.taler.net/<repository> 327 328 For an existing checkout, this can be done by editing the ``.git/config`` file. 329 330 The server is configured to reject all commits that have not been signed with 331 GnuPG. If you do not yet have a GnuPG key, you must create one, as explained 332 in the `GNU Privacy Handbook <https://www.gnupg.org/gph/en/manual/c14.html>`_. 333 You do not need to share the respective public key with us to make commits. 334 However, we recommend that you upload it to key servers, put it on your 335 business card and personally meet with other GNU hackers to have it signed 336 such that others can verify your commits later. 337 338 To sign all commits, you should run 339 340 .. code-block:: console 341 342 $ git config --global commit.gpgsign true 343 344 You can also sign individual commits only by adding the ``-S`` option to the 345 ``git commit`` command. If you accidentally already made commits but forgot 346 to sign them, you can retroactively add signatures using: 347 348 .. code-block:: console 349 350 $ git rebase -S 351 352 353 Whether you commit to a personal branch (recommended: ``dev/$USER/...``), 354 a feature branch or to ``master`` should 355 depend on your level of comfort and the nature of the change. As a general 356 rule, the code in ``master`` must always build and tests should always pass, at 357 least on your own system. However, we all make mistakes and you should expect 358 to receive friendly reminders if your change did not live up to this simple 359 standard. We plan to move to a system where the CI guarantees this invariant 360 in the future. 361 362 In order to keep a linear and clean commits history, we advise to avoid 363 merge commits and instead always rebase your changes before pushing to 364 the ``master`` branch. If you commit and later find out that new commits were 365 pushed, the following command will pull the new commits and rebase yours 366 on top of them. 367 368 .. code-block:: console 369 370 # -S instructs Git to (re)sign your commits 371 $ git pull --rebase -S 372 373 374 375 Observing changes 376 ----------------- 377 378 Every commit to the ``master`` branch of any of our public repositories 379 (and almost all are public) is automatically sent to the 380 gnunet-svn@gnu.org mailinglist. That list is for Git commits only, 381 and must not be used for discussions. It also carries commits from 382 our main dependencies, namely GNUnet and GNU libmicrohttpd. While 383 it can be high volume, the lists is a good way to follow overall 384 development. 385 386 387 Code generator usage policy 388 --------------------------- 389 390 We do neither encourage nor discourage the use of tools for code generation. 391 It is up to the individual developer to decide if a tool is acceptable for 392 a particular task. But of course, we do encourage you to use FLOSS tools 393 and we MUST NOT become dependent on non-free software! That said, if you 394 use tools, you must document their use and in particular satisfy the 395 `NLnet policy on the use of "AI" <https://nlnet.nl/news/2025/20250829-policy-on-use-of-AI.html>`__. 396 397 Specifically, we ask developers to always put generated code into a *separate* 398 Git commit and to include the full prompt in the commit message. Naturally, 399 you may clean up the code generator's output, but then you should do so in 400 separate Git commits (and of course only merge into master/stable after the 401 clean up is complete). But do preserve (not squash!) the commit with the 402 generated code so that it remains documented what the prompts were and which 403 code is generated. This will go a long way to keep code auditors sane! 404 405 406 Communication 407 ------------- 408 409 For public discussions we use the taler@gnu.org mailinglist. All developers 410 should subscribe to the low-volume Taler mailinglist. There are separate 411 low-volume mailinglists for gnunet-developers (@gnu.org) and for libmicrohttpd 412 (@gnu.org). For internal discussions we use https://mattermost.taler.net/ 413 (invitation only, but also archived). 414 415 416 What to put in bootstrap 417 ------------------------ 418 419 Each repository has a ``bootstrap`` script, which contains commands for the 420 developer to run after a repository checkout (i.e., after ``git clone`` or 421 ``git pull``). 422 Typically, this updates and initializes submodules, prepares the tool chain, 423 and runs ``autoreconf``. 424 The last step generates the ``configure`` script, whether for immediate use or 425 for inclusion in the distribution tarball. 426 427 One common submodule is ``contrib/gana``, which pulls from the 428 `GNUnet GANA repository <https://git.gnunet.org/gana.git/>`__. 429 For example, in the 430 `Taler exchange repository <https://git.taler.net/exchange.git>`__, 431 the bootstrap script eventually runs the ``git submodule update --init`` command 432 early on, and later runs script ``./contrib/gana-generate.sh``, which 433 generates files such as ``src/include/taler_signatures.h``. 434 435 Thus, to update that file, you need to: 436 437 - (in GANA repo) Find a suitable (unused) name and number for the Signature 438 Purposes database. 439 440 - Add it to GANA, in ``gnunet-signatures/registry.rec``. 441 (You can check for uniqueness with the ``recfix`` utility.) 442 443 - Commit the change, and push it to the GANA Git repo. 444 445 - (in Taler Repo) Run the ``contrib/gana-latest.sh`` script. 446 447 - Bootstrap, configure, do ``make install``, ``make check``, etc. 448 (Basically, make sure the change does not break anything.) 449 450 - Commit the submodule change, and push it to the Taler exchange Git repo. 451 452 A similar procedure is required for other databases in GANA. 453 See file ``README`` in the various directories for specific instructions. 454 455 456 Debian and Ubuntu Repositories 457 ============================== 458 459 We package our software for Debian and Ubuntu. 460 461 Nightly Repositories 462 -------------------- 463 464 To try the latest, unstable and untested versions of packages, 465 you can add the nightly package sources. 466 467 .. code-block:: shell-session 468 469 # For Debian (trixie) 470 $ curl -sS https://deb.taler.net/apt-nightly/taler-trixie-ci.sources \ 471 | tee /etc/apt/sources.list.d/taler-trixie-nightly.sources 472 473 474 Taler Deployment on gv.taler.net 475 ================================ 476 477 This section describes the GNU Taler deployment on ``gv.taler.net``. ``gv`` 478 is our server at BFH. It hosts the Git repositories, Web sites, CI and other 479 services. Developers can receive an SSH account and e-mail alias for the 480 system, you should contact Javier, Christian or Florian. As with Git, ask 481 your primary team contact for shell access if you think you need it. 482 483 484 DNS 485 --- 486 487 DNS records for taler.net are controlled by the GNU Taler maintainers, 488 specifically Christian and Florian, and our system administrator, Javier. If 489 you need a sub-domain to be added, please contact one of them. 490 491 492 User Acccounts 493 -------------- 494 495 On ``gv.taler.net``, there are three system users that are set up to 496 serve Taler on the Internet: 497 498 - ``head``: serves ``*.head.taler.net`` and gets automatically 499 built by Buildbot every 2 hours from the ``sandcastle-ng.git``. 500 Master key may be reset occasionally 501 502 - ``taler-test``: serves ``*.test.taler.net`` and does *NOT* get 503 automatically built, and runs more recent tags and/or unreleased 504 versions of Taler components. Master key may be reset 505 occasionally. 506 507 - ``demo``: serves ``*.demo.taler.net``. Never automatically built. 508 Master key is retained. 509 510 Demo Upgrade Procedure 511 ====================== 512 513 #. Login as the ``demo`` user on ``gv.taler.net``. 514 #. Pull the latest ``sandcastle-ng.git`` code in checkout at ``$HOME/sandcastle-ng``. 515 #. Run ``systemctl --user restart container-taler-sandcastle-demo.service`` 516 #. Refer to the sandcastle-ng README (https://git.taler.net/sandcastle-ng.git/about/) 517 for more info. 518 519 520 Upgrading the ``demo`` environment should be done with care, and ideally be 521 coordinated on the mailing list before. It is our goal for ``demo`` to always 522 run a "working version" that is compatible with various published wallets. 523 Please use the :doc:`demo upgrade checklist <../checklists/checklist-demo-upgrade>` to make 524 sure everything is working. 525 Nginx is already configured to reach the services as exported by the user unit. 526 527 528 Tagging components 529 ------------------ 530 531 All Taler components must be tagged with git before they are deployed on the 532 ``demo`` environment, using a tag of the following form: 533 534 .. code-block:: none 535 536 demo-YYYY-MM-DD-SS 537 YYYY = year 538 MM = month 539 DD = day 540 SS = serial 541 542 Environments and Builders on taler.net 543 ====================================== 544 545 Buildbot implementation 546 ----------------------- 547 548 GNU Taler uses a buildbot implementation (front end at https://buildbot.taler.net) to manage continuous integration. Buildbot documentation is at https://docs.buildbot.net/. 549 550 Here are some highlights: 551 552 - The WORKER is the config that that lives on a shell account on a localhost (taler.net), where this host has buildbot-worker installed. The WORKER executes the commands that perform all end-functions of buildbot. 553 554 - The WORKER running buildbot-worker receives these commands by authenticating and communicating with the buildbot server using parameters that were specified when the worker was created in that shell account with the ``buildbot-worker`` command. 555 556 - The buildbot server's master.cfg file contains FACTORY declarations which specify the commands that the WORKER will run on localhost. 557 558 - The FACTORY is tied to the WORKER in master.cfg by a BUILDER. 559 560 - The master.cfg also allows for SCHEDULER that defines how and when the BUILDER is executed. 561 562 - Our master.cfg file is checked into git, and then periodically updated on a particular account on taler.net (ask Christian for access if needed). Do not edit this file directly/locally on taler.net, but check changes into Git. 563 564 565 Best Practices: 566 567 - When creating a new WORKER in the ``master.cfg`` file, leave a comment specifying the server and user account that this WORKER is called from. (At this time, taler.net is the only server used by this implementation, but it's still good practice.) 568 569 - Create a worker from a shell account with this command: ``buildbot-worker create-worker <workername> localhost <username> <password>`` 570 571 Then make sure there is a WORKER defined in master.cfg like: ``worker.Worker("<username>", "<password>")`` 572 573 Test builder 574 ------------ 575 576 This builder (``test-builder``) compiles and starts every Taler component. 577 The associated worker is run by the ``taler-test`` Gv user, via the SystemD 578 unit ``buildbot-worker-taler``. The following commands start/stop/restart 579 the worker: 580 581 .. code-block:: 582 583 systemctl --user start buildbot-worker-taler 584 systemctl --user stop buildbot-worker-taler 585 systemctl --user restart buildbot-worker-taler 586 587 .. note:: 588 the mentioned unit file can be found at ``deployment.git/systemd-services/`` 589 590 Wallet builder 591 -------------- 592 593 This builder (``wallet-builder``) compiles every Taler component 594 and runs the wallet integration tests. The associated worker is 595 run by the ``walletbuilder`` Gv user, via the SystemD unit ``buildbot-worker-wallet``. 596 The following commands start/stop/restart the worker: 597 598 .. code-block:: 599 600 systemctl --user start buildbot-worker-wallet 601 systemctl --user stop buildbot-worker-wallet 602 systemctl --user restart buildbot-worker-wallet 603 604 .. note:: 605 the mentioned unit file can be found at ``deployment.git/systemd-services/`` 606 607 Documentation Builder 608 --------------------- 609 610 All the Taler documentation is built by the user ``docbuilder`` that 611 runs a Buildbot worker. The following commands set the ``docbuilder`` up, 612 starting with an empty home directory. 613 614 .. code-block:: console 615 616 # Log-in as the 'docbuilder' user. 617 618 $ cd $HOME 619 $ git clone git://git.taler.net/deployment 620 $ ./deployment/bootstrap-docbuilder 621 622 # If the previous step worked, the setup is 623 # complete and the Buildbot worker can be started. 624 625 $ buildbot-worker start worker/ 626 627 628 Website Builder 629 --------------- 630 631 632 Taler Websites, ``www.taler.net`` and ``stage.taler.net``, are built by the 633 user ``taler-websites`` by the means of a Buildbot worker. The following 634 commands set the ``taler-websites`` up, starting with an empty home directory. 635 636 .. code-block:: console 637 638 # Log-in as the 'taler-websites' user. 639 640 $ cd $HOME 641 $ git clone git://git.taler.net/deployment 642 $ ./deployment/bootstrap-sitesbuilder 643 644 # If the previous step worked, the setup is 645 # complete and the Buildbot worker can be started. 646 647 $ buildbot-worker start worker/ 648 649 650 Code coverage 651 ------------- 652 653 Code coverage tests are run by the ``lcovworker`` user, and are also driven 654 by Buildbot. 655 656 .. code-block:: console 657 658 # Log-in as the 'lcovworker' user. 659 660 $ cd $HOME 661 $ git clone git://git.taler.net/deployment 662 $ ./deployment/bootstrap-taler lcov 663 664 # If the previous step worked, the setup is 665 # complete and the Buildbot worker can be started. 666 667 $ buildbot-worker start worker/ 668 669 The results are then published at ``https://lcov.taler.net/``. 670 671 Producing auditor reports 672 ------------------------- 673 674 Both 'test' and 'demo' setups get their auditor reports compiled 675 by a Buildbot worker. The following steps get the reports compiler 676 prepared. 677 678 .. code-block:: console 679 680 # Log-in as <env>-auditor, with <env> being either 'test' or 'demo' 681 682 $ git clone git://git.taler.net/deployment 683 $ ./deployment/buildbot/bootstrap-scripts/prepare-auditorreporter <env> 684 685 # If the previous steps worked, then it should suffice to start 686 # the worker, with: 687 688 $ buildbot-worker start worker/ 689 690 691 .. _DatabaseVersioning: 692 693 Database schema versioning 694 -------------------------- 695 696 The PostgreSQL databases of the exchange, auditor, and merchant are versioned. 697 See the ``versioning.sql`` file in the respective directory for documentation. 698 699 Every set of changes to the database schema must be stored in a new 700 versioned SQL script. The scripts must have contiguous numbers. After 701 any release (or version being deployed to a production or staging 702 environment), existing scripts MUST be immutable. 703 704 Developers and operators MUST NOT make changes to database schema 705 outside of this versioning. All tables of a GNU Taler component should live in their own schema. 706 707 Runtime stored procedures are loaded from the component's generated 708 ``procedures.sql`` after schema patches. Updating a runtime procedure does 709 not itself advance the schema-patch version; table, index, constraint, and 710 data migrations still require a new numbered script. Component database 711 initialization tools are the supported way to apply both the pending patches 712 and the current procedure definitions. 713 714 715 QA Plans 716 ======== 717 718 .. include:: ../checklists/qa-1.0.rst 719 720 721 Releases 722 ======== 723 724 .. include:: ../checklists/checklist-release.rst 725 726 Release Process 727 --------------- 728 729 This document describes the process for releasing a new version of the 730 various Taler components to the official GNU mirrors. 731 732 The following components are published on the GNU mirrors 733 734 - taler-exchange (exchange.git) 735 - taler-merchant (merchant.git) 736 - sync (sync.git) 737 - taler-mdb (taler-mdb.git) 738 - libeufin (libeufin.git) 739 - challenger (challenger.git) 740 - wallet-core (wallet-core.git) 741 742 Tagging 743 ------- 744 745 Tag releases with an **annotated** commit, like 746 747 .. code-block:: console 748 749 $ git tag -a v0.1.0 -m "Official release v0.1.0" 750 $ git push origin v0.1.0 751 752 Prebuilt artifact policy 753 ------------------------ 754 755 Prebuilt artifacts are stored separately from source, normally on an orphan 756 ``prebuilt`` branch, and are consumed through a Git submodule pinned to an 757 exact commit. Projects producing such artifacts MUST provide repeatable 758 Makefile targets that build the artifact, prepare the prebuilt worktree, and 759 install the result in that worktree. A consumer's ``bootstrap`` script MUST 760 initialize the submodule and SHOULD fail gracefully when an obsolete prebuilt 761 commit is no longer available. 762 763 Artifacts use the layout ``COMPONENT/VERSION/``. ``VERSION`` MUST match both 764 the component version and a source tag. Every revision intended for consumers 765 MUST have a ``prebuilt-SERIAL`` tag; a ``COMPONENT/VERSION`` tag MAY additionally 766 identify the component-specific payload. Consumers MUST pin a tagged commit 767 and SHOULD use sparse checkout. Source, build instructions, and dependency 768 versions must remain sufficient to reproduce the artifact offline. 769 770 771 Database for tests 772 ------------------ 773 774 For tests in the exchange and merchant to run, make sure that a database 775 *talercheck* is accessible by *$USER*. Otherwise tests involving the 776 database logic are skipped. 777 778 .. include:: ../frags/db-stores-sensitive-data.rst 779 780 Exchange, merchant 781 ------------------ 782 783 Set the version in ``configure.ac``. The commit being tagged should be 784 the change of the version. 785 786 Tag the current GANA version that works with the exchange and merchant and 787 checkout that tag of gana.git (instead of master). Otherwise, if there are 788 incompatible changes in GANA (like removed symbols), old builds could break. 789 790 Update the Texinfo documentation using the files from docs.git: 791 792 .. code-block:: console 793 794 # Get the latest documentation repository 795 $ cd $GIT/docs 796 $ git pull 797 $ make texinfo 798 # The *.texi files are now in _build/texinfo 799 # 800 # This checks out the prebuilt branch in the prebuilt directory 801 $ git worktree add prebuilt prebuilt 802 $ cd prebuilt 803 # Copy the pre-built documentation into the prebuilt directory 804 $ cp -r ../_build/texinfo . 805 # Push and commit to branch 806 $ git commit -a -S -m "updating texinfo" 807 $ git status 808 # Verify that all files that should be tracked are tracked, 809 # new files will have to be added to the Makefile.am in 810 # exchange.git as well! 811 $ git push 812 # Remember $REVISION of commit 813 # 814 # Go to exchange 815 $ cd $GIT/exchange/doc/prebuilt 816 # Update submodule to point to latest commit 817 $ git checkout $REVISION 818 819 Finally, the Automake ``Makefile.am`` files may have to be adjusted to 820 include new ``*.texi`` files or images. 821 822 For bootstrap, you will need to install 823 `GNU Recutils <https://www.gnu.org/software/recutils/>`_. 824 825 For the exchange test cases to pass, ``make install`` must be run first. 826 Without it, test cases will fail because plugins can't be located. 827 828 .. code-block:: console 829 830 $ ./bootstrap 831 $ ./configure # add required options for your system 832 $ make dist 833 $ tar -xf taler-$COMPONENT-$VERSION.tar.gz 834 $ cd taler-$COMPONENT-$VERSION 835 $ make install check 836 837 Wallet WebExtension 838 ------------------- 839 840 The version of the wallet is in *manifest.json*. The ``version_name`` 841 should be adjusted, and *version* should be increased independently on 842 every upload to the WebStore. 843 844 .. code-block:: console 845 846 $ ./configure 847 $ make dist 848 849 Upload to GNU mirrors 850 --------------------- 851 852 See https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads 853 854 Directive file: 855 856 .. code-block:: none 857 858 version: 1.2 859 directory: taler 860 filename: taler-exchange-0.1.0.tar.gz 861 symlink: taler-exchange-0.1.0.tar.gz taler-exchange-latest.tar.gz 862 863 Upload the files in **binary mode** to the ftp servers. 864 865 866 Creating Debian packages 867 ------------------------ 868 869 Our general setup is based on 870 https://wiki.debian.org/DebianRepository/SetupWithReprepro 871 872 First, update at least the version of the Debian package in 873 debian/changelog, and then run: 874 875 .. code-block:: bash 876 877 $ dpkg-buildpackage -rfakeroot -b -uc -us 878 879 in the respective source directory (GNUnet, exchange, merchant) to create the 880 ``.deb`` files. Note that they will be created in the parent directory. This 881 can be done on gv.taler.net, or on another (secure) machine. 882 Actual release builds should be done via the Docker images 883 that can be found in ``deployment.git`` under packaging. 884 885 On ``gv``, we use the ``aptbuilder`` user to manage the reprepro repository. 886 887 Next, the ``*.deb`` files should be copied to gv.taler.net, say to 888 ``/home/aptbuilder/incoming``. Then, run 889 890 .. code-block:: bash 891 892 # cd /home/aptbuilder/apt 893 # reprepro includedeb bullseye ~/incoming/*.deb 894 895 to import all Debian files from ``~/incoming/`` into the ``bullseye`` 896 distribution. If Debian packages were build against other distributions, 897 reprepro may need to be first configured for those and the import command 898 updated accordingly. 899 900 Finally, make sure to clean up ``~/incoming/`` (by deleting the 901 now imported ``*.deb`` files). 902 903 904 905 Continuous integration 906 ====================== 907 908 CI is done with Buildbot (https://buildbot.net/), and builds are 909 triggered by the means of Git hooks. The results are published at 910 https://buildbot.taler.net/ . 911 912 In order to avoid downtimes, CI uses a "blue/green" deployment 913 technique. In detail, there are two users building code on the system, 914 the "green" and the "blue" user; and at any given time, one is running 915 Taler services and the other one is either building the code or waiting 916 for that. 917 918 There is also the possibility to trigger builds manually, but this is 919 only reserved to "admin" users. 920 921 Each repository owns its CI definition under ``contrib/ci``. Jobs are 922 directories named ``contrib/ci/jobs/N-NAME`` and MUST contain ``job.sh``; 923 their numeric prefix determines execution order. Optional job configuration 924 is stored as ``config.ini``. Unless every job selects its own container, the 925 repository MUST provide ``contrib/ci/Containerfile``. Projects SHOULD provide 926 at least separate build and test jobs. 927 928 The repository's ``contrib/ci/ci.sh NAME`` entry point runs an individual job 929 locally in the same containerized environment used by Buildbot. CI behavior 930 belongs in the component repository rather than in a central Buildbot-only 931 configuration. Cross-repository pipelines are separate builders because they 932 require several source trees and have different triggering requirements. 933 934 935 Dynamic form metadata 936 ===================== 937 938 Taler Web UIs that consume backend-provided form descriptions use a versioned 939 ``FormMetadata`` envelope. It contains a stable form ``id``, human-readable 940 ``label`` and optional ``description``, numeric ``version``, and a ``config`` 941 object. JSON configurations use one of two layouts: 942 943 * ``single-column`` with an ordered ``fields`` list; or 944 * ``double-column`` with an ordered ``sections`` list, where each section has a 945 title, optional description, and fields. 946 947 Fields are discriminated by their ``type``. Currently supported field classes 948 include amount, boolean, date, duration, one- and multi-select, text, 949 multi-line text, and toggle controls. Consumers MUST reject unknown or 950 malformed structures through the shared codec instead of interpreting them as 951 arbitrary HTML. The version belongs to the form definition and must change 952 when an incompatible metadata interpretation is introduced. 953 954 955 Internationalisation 956 ==================== 957 958 Internationalisation (a.k.a "translation") is handled using text-based 959 localization files named PO (Portable Object) holding pairs of original and 960 translated strings. 961 962 .. include:: dictionary.rst 963 :start-line: 4 964 965 iOS localization uses the checked-in ``Localizable.xcstrings`` catalogs as the 966 application source and checked-in ``XLIFF/*.xliff`` files as the Weblate 967 interchange representation. The XLIFF ``original`` attributes identify the 968 source ``.xcstrings`` catalogs. Translation updates must round-trip through 969 Xcode's XLIFF import/export support, preserve message identifiers and 970 placeholders, and be followed by a full application build. The older proposed 971 ``pogen`` conversion through PO files is not the deployed iOS workflow. 972 973 974 iOS Apps 975 ======== 976 977 .. _Build-iOS-from-source: 978 979 Building Taler Wallet for iOS from source 980 ----------------------------------------- 981 982 The GNU Taler Wallet iOS app is in 983 `the official Git repository <https://git.taler.net/taler-ios.git>`__. 984 985 Compatibility 986 ^^^^^^^^^^^^^ 987 988 The minimum version of iOS supported is 15.0. 989 This app runs on all iPhone models at least as new as the iPhone 6S. 990 991 992 Building 993 ^^^^^^^^ 994 995 Before building the iOS wallet, you must first checkout the 996 `quickjs-tart repo <https://git.taler.net/quickjs-tart.git>`__ 997 and the 998 `wallet-core repo <https://git.taler.net/wallet-core.git>`__. 999 1000 Have all 3 local repos (wallet-core, quickjs-tart, and this one) adjacent at 1001 the same level (e.g. in a "GNU_Taler" folder) 1002 Taler.xcworkspace expects the QuickJS framework sub-project to be at 1003 ``../quickjs-tart/QuickJS-rt.xcodeproj``. 1004 1005 Build wallet-core first: 1006 1007 .. code-block:: shell-session 1008 1009 $ cd wallet-core 1010 $ make embedded 1011 $ open packages/taler-wallet-embedded/dist 1012 1013 then drag or move its product "taler-wallet-core-qjs.mjs" 1014 into your quickjs-tart folder right at the top level. 1015 1016 Open Taler.xcworkspace, and set scheme / target to Taler_Wallet. Build&run... 1017 1018 Don't open QuickJS-rt.xcodeproj or TalerWallet.xcodeproj and build anything 1019 there - all needed libraries and frameworks will be built automatically from 1020 Taler.xcworkspace. 1021 1022 1023 Android Apps 1024 ============ 1025 1026 Android App Nightly Builds 1027 -------------------------- 1028 1029 There are currently three Android apps in 1030 `the official Git repository <https://git.taler.net/taler-android.git>`__: 1031 1032 * Wallet 1033 [`CI <https://git.taler.net/taler-android.git/tree/wallet/.gitlab-ci.yml>`__] 1034 * Merchant PoS Terminal 1035 [`CI <https://git.taler.net/taler-android.git/tree/merchant-terminal/.gitlab-ci.yml>`__] 1036 * Cashier 1037 [`CI <https://git.taler.net/taler-android.git/tree/cashier/.gitlab-ci.yml>`__] 1038 1039 Their git repositories are `mirrored at Gitlab <https://gitlab.com/gnu-taler/taler-android>`__ 1040 to utilize their CI 1041 and `F-Droid <https://f-droid.org>`_'s Gitlab integration 1042 to `publish automatic nightly builds <https://f-droid.org/docs/Publishing_Nightly_Builds/>`_ 1043 for each change on the ``master`` branch. 1044 1045 All three apps publish their builds to the same F-Droid nightly repository 1046 (which is stored as a git repository): 1047 https://gitlab.com/gnu-taler/fdroid-repo-nightly 1048 1049 You can download the APK files directly from that repository 1050 or add it to the F-Droid app for automatic updates 1051 by clicking the following link (on the phone that has F-Droid installed). 1052 1053 `GNU Taler Nightly F-Droid Repository <fdroidrepos://gnu-taler.gitlab.io/fdroid-repo-nightly/fdroid/repo?fingerprint=55F8A24F97FAB7B0960016AF393B7E57E7A0B13C2D2D36BAC50E1205923A7843>`_ 1054 1055 .. note:: 1056 Nightly apps can be installed alongside official releases 1057 and thus are meant **only for testing purposes**. 1058 Use at your own risk! 1059 1060 .. _Build-apps-from-source: 1061 1062 Building apps from source 1063 ------------------------- 1064 1065 Note that this guide is different from other guides for building Android apps, 1066 because it does not require you to run non-free software. 1067 It uses the Merchant PoS Terminal as an example, but works as well for the other apps 1068 if you replace ``merchant-terminal`` with ``wallet`` or ``cashier``. 1069 1070 First, ensure that you have the required dependencies installed: 1071 1072 * Java Development Kit 8 or higher (default-jdk-headless) 1073 * git 1074 * unzip 1075 1076 Then you can get the app's source code using git: 1077 1078 .. code-block:: console 1079 1080 # Start by cloning the Android git repository 1081 $ git clone https://git.taler.net/taler-android.git 1082 1083 # Change into the directory of the cloned repository 1084 $ cd taler-android 1085 1086 # Find out which Android SDK version you will need 1087 $ grep -i compileSdkVersion merchant-terminal/build.gradle 1088 1089 The last command will return something like ``compileSdkVersion 29``. 1090 So visit the `Android Rebuilds <http://android-rebuilds.beuc.net/>`_ project 1091 and look for that version of the Android SDK there. 1092 If the SDK version is not yet available as a free rebuild, 1093 you can try to lower the ``compileSdkVersion`` in the app's ``merchant-terminal/build.gradle`` file. 1094 Note that this might break things 1095 or require you to also lower other versions such as ``targetSdkVersion``. 1096 1097 In our example, the version is ``29`` which is available, 1098 so download the "SDK Platform" package of "Android 10.0.0 (API 29)" 1099 and unpack it: 1100 1101 .. code-block:: console 1102 1103 # Change into the directory that contains your downloaded SDK 1104 $ cd $HOME 1105 1106 # Unpack/extract the Android SDK 1107 $ unzip android-sdk_eng.10.0.0_r14_linux-x86.zip 1108 1109 # Tell the build system where to find the SDK 1110 $ export ANDROID_SDK_ROOT="$HOME/android-sdk_eng.10.0.0_r14_linux-x86" 1111 1112 # Change into the directory of the cloned repository 1113 $ cd taler-android 1114 1115 # Build the merchant-terminal app 1116 $ ./gradlew :merchant-terminal:assembleRelease 1117 1118 If you get an error message complaining about build-tools 1119 1120 > Failed to install the following Android SDK packages as some licences have not been accepted. 1121 build-tools;29.0.3 Android SDK Build-Tools 29.0.3 1122 1123 you can try changing the ``buildToolsVersion`` in the app's ``merchant-terminal/build.gradle`` file 1124 to the latest "Android SDK build tools" version supported by the Android Rebuilds project. 1125 1126 After the build finished successfully, 1127 you will find your APK in ``merchant-terminal/build/outputs/apk/release/``. 1128 1129 Update translations 1130 ------------------- 1131 1132 Translations are managed with Taler's weblate instance: 1133 https://weblate.taler.net/projects/gnu-taler/ 1134 1135 To update translations, enter the taler-android git repository 1136 and ensure that the weblate remote exists: 1137 1138 .. code-block:: console 1139 1140 $ git config -l | grep weblate 1141 1142 If it does not yet exist (empty output), you can add it like this: 1143 1144 .. code-block:: console 1145 1146 $ git remote add weblate https://weblate.taler.net/git/gnu-taler/wallet-android/ 1147 1148 Then you can merge in translations commit from the weblate remote: 1149 1150 .. code-block:: console 1151 1152 # ensure you have latest version 1153 $ git fetch weblate 1154 1155 # merge in translation commits 1156 $ git merge weblate/master 1157 1158 Afterwards, build the entire project from source and test the UI 1159 to ensure that no erroneous translations (missing placeholders) are breaking things. 1160 1161 Release process 1162 --------------- 1163 1164 After extensive testing, the code making up a new release should get a signed git tag. 1165 The current tag format is: 1166 1167 * cashier-$VERSION 1168 * pos-$VERSION 1169 * wallet-$VERSION (where $VERSION has a v prefix) 1170 1171 .. code-block:: console 1172 1173 $ git tag -s $APP-$VERSION 1174 1175 F-Droid 1176 ^^^^^^^ 1177 Nightly builds get published automatically (see above) after pushing code to the official repo. 1178 Actual releases get picked up by F-Droid's official repository via git tags. 1179 So ensure that all releases get tagged properly. 1180 1181 Some information for F-Droid official repository debugging: 1182 1183 * Wallet: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.wallet.fdroid.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.wallet.fdroid/lastbuild>`__] 1184 * Cashier: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.cashier.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.cashier/lastbuild>`__] 1185 * PoS: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.merchantpos.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.merchantpos/lastbuild>`__] 1186 1187 Google Play 1188 ^^^^^^^^^^^ 1189 Google Play uploads are managed via `Fastlane <https://docs.fastlane.tools/getting-started/android/setup/>`__. 1190 Before proceeding, ensure that this is properly set up 1191 and that you have access to the Google Play API. 1192 1193 It is important to have access to the signing keys and Google Play access keys 1194 (JSON) and to ensure that the following environment variables are set 1195 correctly and made available to Fastlane: 1196 1197 .. code-block:: bash 1198 1199 TALER_KEYSTORE_PATH= 1200 TALER_KEYSTORE_PASS= 1201 TALER_KEYSTORE_WALLET_ALIAS= 1202 TALER_KEYSTORE_WALLET_PASS= 1203 TALER_KEYSTORE_POS_ALIAS= 1204 TALER_KEYSTORE_POS_PASS= 1205 TALER_KEYSTORE_CASHIER_ALIAS= 1206 TALER_KEYSTORE_CASHIER_PASS= 1207 TALER_JSON_KEY_FILE= 1208 1209 To release an app, enter into its respective folder and run fastlane: 1210 1211 .. code-block:: console 1212 1213 $ bundle exec fastlane 1214 1215 Then select the deploy option. 1216 1217 All uploads are going to the beta track by default. These can be promoted to 1218 production later or immediately after upload if you feel daring. It is also 1219 important to bump the version and build code with every release. 1220 1221 .. _Code-coverage: 1222 1223 Code Coverage 1224 ============= 1225 1226 Code coverage is done with the Gcov / Lcov 1227 (http://ltp.sourceforge.net/coverage/lcov.php) combo, and it is run 1228 nightly (once a day) by a Buildbot worker. The coverage results are 1229 then published at https://lcov.taler.net/ . 1230 1231 1232 Coding Conventions 1233 ================== 1234 1235 GNU Taler is developed primarily in C, Kotlin, Python, Swift and TypeScript. 1236 1237 Components written in C 1238 ----------------------- 1239 1240 These are the general coding style rules for Taler. 1241 1242 * Baseline rules are to follow GNU guidelines, modified or extended 1243 by the GNUnet style: https://docs.gnunet.org/handbook/gnunet.html#Coding-style 1244 1245 Naming conventions 1246 ^^^^^^^^^^^^^^^^^^ 1247 1248 * include files (very similar to GNUnet): 1249 1250 * if installed, must start with "``taler_``" (exception: platform.h), 1251 and MUST live in src/include/ 1252 * if NOT installed, must NOT start with "``taler_``" and 1253 MUST NOT live in src/include/ and 1254 SHOULD NOT be included from outside of their own directory 1255 * end in "_lib" for "simple" libraries 1256 * end in "_plugin" for plugins 1257 * end in "_service" for libraries accessing a service, i.e. the exchange 1258 1259 * binaries: 1260 1261 * taler-exchange-xxx: exchange programs 1262 * taler-merchant-xxx: merchant programs (demos) 1263 * taler-wallet-xxx: wallet programs 1264 * plugins should be libtaler_plugin_xxx_yyy.so: plugin yyy for API xxx 1265 * libtalerxxx: library for API xxx 1266 1267 * logging 1268 1269 * tools use their full name in GNUNET_log_setup 1270 (i.e. 'taler-exchange-offline') and log using plain 'GNUNET_log'. 1271 * pure libraries (without associated service) use 'GNUNET_log_from' 1272 with the component set to their library name (without lib or '.so'), 1273 which should also be their directory name (i.e. 'util') 1274 * plugin libraries (without associated service) use 'GNUNET_log_from' 1275 with the component set to their type and plugin name (without lib or '.so'), 1276 which should also be their directory name (i.e. 'exchangedb-postgres') 1277 * libraries with associated service) use 'GNUNET_log_from' 1278 with the name of the service, which should also be their 1279 directory name (i.e. 'exchange') 1280 * for tools with ``-l LOGFILE``, its absence means write logs to stderr 1281 1282 * configuration 1283 1284 * same rules as for GNUnet 1285 1286 * exported symbols 1287 1288 * must start with TALER_[SUBSYSTEMNAME]_ where SUBSYSTEMNAME 1289 MUST match the subdirectory of src/ in which the symbol is defined 1290 * from libtalerutil start just with ``TALER_``, without subsystemname 1291 * if scope is ONE binary and symbols are not in a shared library, 1292 use binary-specific prefix (such as TMH = taler-exchange-httpd) for 1293 globals, possibly followed by the subsystem (TMH_DB_xxx). 1294 1295 * structs: 1296 1297 * structs that are 'packed' and do not contain pointers and are 1298 thus suitable for hashing or similar operations are distinguished 1299 by adding a "P" at the end of the name. (NEW) Note that this 1300 convention does not hold for the GNUnet-structs (yet). 1301 * structs that are used with a purpose for signatures, additionally 1302 get an "S" at the end of the name. 1303 1304 * private (library-internal) symbols (including structs and macros) 1305 1306 * must not start with ``TALER_`` or any other prefix 1307 1308 * testcases 1309 1310 * must be called "test_module-under-test_case-description.c" 1311 1312 * performance tests 1313 1314 * must be called "perf_module-under-test_case-description.c" 1315 1316 Shell Scripts 1317 ------------- 1318 1319 Shell scripts should be avoided if at all possible. The only permissible uses of shell scripts 1320 in GNU Taler are: 1321 1322 * Trivial invocation of other commands. 1323 * Scripts for compatibility (e.g. ``./configure``) that must run on 1324 as many systems as possible. 1325 1326 When shell scripts are used, they ``MUST`` begin with the following ``set`` command: 1327 1328 .. code-block:: console 1329 1330 # Make the shell fail on undefined variables and 1331 # commands with non-zero exit status. 1332 $ set -eu 1333 1334 Kotlin 1335 ------ 1336 1337 We so far have no specific guidelines, please follow best practices 1338 for the language. 1339 1340 1341 Python 1342 ------ 1343 1344 Supported Python Versions 1345 ^^^^^^^^^^^^^^^^^^^^^^^^^ 1346 1347 Python code should be written and built against version 3.7 of Python. 1348 1349 Style 1350 ^^^^^ 1351 1352 We use `yapf <https://github.com/google/yapf>`_ to reformat the 1353 code to conform to our style instructions. 1354 A reusable yapf style file can be found in ``build-common``, 1355 which is intended to be used as a git submodule. 1356 1357 Python for Scripting 1358 ^^^^^^^^^^^^^^^^^^^^ 1359 1360 When using Python for writing small utilities, the following libraries 1361 are useful: 1362 1363 * ``click`` for argument parsing (should be preferred over argparse) 1364 * ``pathlib`` for path manipulation (part of the standard library) 1365 * ``subprocess`` for "shelling out" to other programs. Prefer ``subprocess.run`` 1366 over the older APIs. 1367 1368 1369 Swift 1370 ----- 1371 1372 Please follow best practices for the language. 1373 1374 1375 TypeScript 1376 ---------- 1377 1378 Please follow best practices for the language. 1379 1380 1381 Testing library 1382 =============== 1383 1384 This chapter is a VERY ABSTRACT description of how testing is 1385 implemented in Taler, and in NO WAY wants to substitute the reading of 1386 the actual source code by the user. 1387 1388 In Taler, a test case is an array of ``struct TALER_TESTING_Command``, 1389 informally referred to as ``CMD``, that is iteratively executed by the 1390 testing interpreter. This latter is transparently initiated by the 1391 testing library. 1392 1393 However, the developer does not have to define CMDs manually, but 1394 rather call the proper constructor provided by the library. For example, 1395 if a CMD is supposed to test feature ``x``, then the library would 1396 provide the ``TALER_TESTING_cmd_x ()`` constructor for it. Obviously, 1397 each constructor has its own particular arguments that make sense to 1398 test ``x``, and all constructors are thoroughly commented within the 1399 source code. 1400 1401 Internally, each CMD has two methods: ``run ()`` and ``cleanup ()``. The 1402 former contains the main logic to test feature ``x``, whereas the latter 1403 cleans the memory up after execution. 1404 1405 In a test life, each CMD needs some internal state, made by values it 1406 keeps in memory. Often, the test has to *share* those values with other 1407 CMDs: for example, CMD1 may create some key material and CMD2 needs this 1408 key material to encrypt data. 1409 1410 The offering of internal values from CMD1 to CMD2 is made by *traits*. A 1411 trait is a ``struct TALER_TESTING_Trait``, and each CMD contains an array 1412 of traits, that it offers via the public trait interface to other 1413 commands. The definition and filling of such array happens transparently 1414 to the test developer. 1415 1416 For example, the following example shows how CMD2 takes an amount object 1417 offered by CMD1 via the trait interface. 1418 1419 Note: the main interpreter and the most part of CMDs and traits are 1420 hosted inside the exchange codebase, but nothing prevents the developer 1421 from implementing new CMDs and traits within other codebases. 1422 1423 .. code-block:: c 1424 1425 /* Without loss of generality, let's consider the 1426 * following logic to exist inside the run() method of CMD1 */ 1427 ... 1428 1429 struct TALER_Amount *a; 1430 /** 1431 * the second argument (0) points to the first amount object offered, 1432 * in case multiple are available. 1433 */ 1434 if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a)) 1435 return GNUNET_SYSERR; 1436 ... 1437 1438 use(a); /* 'a' points straight into the internal state of CMD2 */ 1439 1440 In the Taler realm, there is also the possibility to alter the behaviour 1441 of supposedly well-behaved components. This is needed when, for example, 1442 we want the exchange to return some corrupted signature in order to 1443 check if the merchant backend detects it. 1444 1445 This alteration is accomplished by another service called *twister*. The 1446 twister acts as a proxy between service A and B, and can be programmed 1447 to tamper with the data exchanged by A and B. 1448 1449 Please refer to the Twister codebase (under the ``test`` directory) in 1450 order to see how to configure it. 1451 1452 1453 User-Facing Terminology 1454 ======================= 1455 1456 This section contains terminology that should be used and that should not be 1457 used in the user interface and help materials. 1458 1459 Terms to Avoid 1460 -------------- 1461 1462 Refreshing 1463 Refreshing is the internal technical terminology for the protocol to 1464 give change for partially spent coins 1465 1466 **Use instead**: "Obtaining change" 1467 1468 Charge 1469 Charge has two opposite meanings (charge to a credit card vs. charge a battery). 1470 This can confuse users. 1471 1472 **Use instead**: "Obtain", "Credit", "Debit", "Withdraw", "Top up" 1473 1474 Coin 1475 Coins are an internal construct, the user should never be aware that their balance 1476 is represented by coins of different denominations. 1477 1478 **Use instead**: "(Digital) Cash" or "(Wallet) Balance" 1479 1480 Consumer 1481 Has bad connotation of consumption. 1482 1483 **Use instead**: Customer or user. 1484 1485 Proposal 1486 The term used to describe the process of the merchant facilitating the download 1487 of the signed contract terms for an order. 1488 1489 **Avoid**. Generally events that relate to proposal downloads 1490 should not be shown to normal users, only developers. Instead, use 1491 "communication with merchant failed" if a proposed order can't be downloaded. 1492 1493 Anonymous E-Cash 1494 Should be generally avoided, since Taler is only anonymous for 1495 the customer. Also some people are scared of anonymity (which as 1496 a term is also way too absolute, as anonymity is hardly ever perfect). 1497 1498 **Use instead**: "Privacy-preserving", "Privacy-friendly" 1499 1500 Payment Replay 1501 The process of proving to the merchant that the customer is entitled 1502 to view a digital product again, as they already paid for it. 1503 1504 **Use instead**: In the event history, "re-activated digital content purchase" 1505 could be used. (FIXME: this is still not nice.) 1506 1507 Session ID 1508 See Payment Replay. 1509 1510 Order 1511 Too ambiguous in the wallet. 1512 1513 **Use instead**: Purchase 1514 1515 Fulfillment URL 1516 URL that serves the digital content that the user purchased 1517 with their payment. Can also be something like a donation receipt. 1518 1519 Donau 1520 Developer-internal name for the tax authority component. 1521 1522 **Use instead**: Tax authority 1523 1524 Terms to Use 1525 ------------ 1526 1527 Auditor 1528 Regulatory entity that certifies exchanges and oversees their operation. 1529 1530 Exchange Operator 1531 The entity/service that gives out digital cash in exchange for some 1532 other means of payment. 1533 1534 In some contexts, using "Issuer" could also be appropriate. 1535 When showing a balance breakdown, 1536 we can say "100 Eur (issued by exchange.euro.taler.net)". 1537 Sometimes we may also use the more generic term "Payment Service Provider" 1538 when the concept of an "Exchange" is still unclear to the reader. 1539 1540 Refund 1541 A refund is given by a merchant to the customer (rather the customer's wallet) 1542 and "undoes" a previous payment operation. 1543 1544 Payment 1545 The act of sending digital cash to a merchant to pay for an order. 1546 1547 Purchase 1548 Used to refer to the "result" of a payment, as in "view purchase". 1549 Use sparingly, as the word doesn't fit for all payments, such as donations. 1550 1551 Contract Terms 1552 Partially machine-readable representation of the merchant's obligation after the 1553 customer makes a payment. 1554 1555 Merchant 1556 Party that receives a payment. 1557 1558 Wallet 1559 Also "Taler Wallet". Software component that manages the user's digital cash 1560 and payments. 1561 1562 1563 Developer Glossary 1564 ================== 1565 1566 This glossary is meant for developers. It contains some terms that we usually do not 1567 use when talking to end users or even system administrators. 1568 1569 .. glossary:: 1570 :sorted: 1571 1572 absolute time 1573 method of keeping time in :term:`GNUnet` where the time is represented 1574 as the number of microseconds since 1.1.1970 (UNIX epoch). Called 1575 absolute time in contrast to :term:`relative time`. 1576 1577 aggregate 1578 the :term:`exchange` combines multiple payments received by the 1579 same :term:`merchant` into one larger :term:`wire transfer` to 1580 the respective merchant's :term:`bank` account 1581 1582 auditor 1583 trusted third party that verifies that the :term:`exchange` is operating correctly 1584 1585 bank 1586 traditional financial service provider who offers 1587 :term:`wire transfers <wire transfer>` between accounts 1588 1589 buyer 1590 individual in control of a Taler :term:`wallet`, usually using it to 1591 :term:`spend` the :term:`coins <coin>` on :term:`contracts <contract>` (see also :term:`customer`). 1592 1593 close 1594 operation an :term:`exchange` performs on a :term:`reserve` that has not been 1595 :term:`emptied <empty>` by :term:`withdraw` operations. When closing a reserve, the 1596 exchange wires the remaining funds back to the customer, minus a :term:`fee` 1597 for closing 1598 1599 customer 1600 individual that directs the buyer (perhaps the same individual) to make a purchase 1601 1602 coin 1603 coins are individual tokens representing a certain amount of value, also known as the :term:`denomination` of the coin 1604 1605 refresh commitment 1606 data that the wallet commits to during the :term:`melt` stage of the 1607 :term:`refresh` protocol where it 1608 has to prove to the :term:`exchange` that it is deriving the :term:`fresh` 1609 coins as specified by the Taler protocol. The commitment is verified 1610 probabilistically (see: :term:`kappa`) during the :term:`reveal` stage. 1611 1612 contract 1613 formal agreement between :term:`merchant` and :term:`customer` specifying the 1614 :term:`contract terms` and signed by the merchant and the :term:`coins <coin>` of the 1615 customer 1616 1617 contract terms 1618 the individual clauses specifying what the buyer is purchasing from the 1619 :term:`merchant` 1620 1621 denomination 1622 unit of currency, specifies both the currency and the face value of a :term:`coin`, 1623 as well as associated fees and validity periods 1624 1625 denomination key 1626 (RSA) key used by the exchange to certify that a given :term:`coin` is valid and of a 1627 particular :term:`denomination` 1628 1629 deposit 1630 operation by which a merchant passes coins to an exchange, expecting the 1631 exchange to credit his bank account in the future using an 1632 :term:`aggregate` :term:`wire transfer` 1633 1634 drain 1635 process by which an exchange operator takes the profits 1636 (from :term:`fees <fee>`) out of the escrow account and moves them into 1637 their regular business account 1638 1639 dirty 1640 a :term:`coin` is dirty if its public key may be known to an entity other than 1641 the customer, thereby creating the danger of some entity being able to 1642 link multiple transactions of coin's owner if the coin is not refreshed 1643 1644 empty 1645 a :term:`reserve` is being emptied when a :term:`wallet` is using the 1646 reserve's private key to :term:`withdraw` coins from it. This reduces 1647 the balance of the reserve. Once the balance reaches zero, we say that 1648 the reserve has been (fully) emptied. Reserves that are not emptied 1649 (which is the normal process) are :term:`closed <close>` by the exchange. 1650 1651 exchange 1652 Taler's payment service operator. Issues electronic coins during 1653 withdrawal and redeems them when they are deposited by merchants 1654 1655 expired 1656 Various operations come with time limits. In particular, denomination keys 1657 come with strict time limits for the various operations involving the 1658 coin issued under the denomination. The most important limit is the 1659 deposit expiration, which specifies until when wallets are allowed to 1660 use the coin in deposit or refreshing operations. There is also a "legal" 1661 expiration, which specifies how long the exchange keeps records beyond the 1662 deposit expiration time. This latter expiration matters for legal disputes 1663 in courts and also creates an upper limit for refreshing operations on 1664 special zombie coin 1665 1666 GNUnet 1667 Codebase of various libraries for a better Internet, some of which 1668 GNU Taler depends upon. 1669 1670 fakebank 1671 implementation of the :term:`bank` API in memory to be used only for test 1672 cases. 1673 1674 fee 1675 an :term:`exchange` charges various fees for its service. The different 1676 fees are specified in the protocol. There are fees per coin for 1677 :term:`withdrawing <withdraw>`, :term:`depositing <deposit>`, :term:`melting <melt>`, and 1678 :term:`refunding <refund>`. Furthermore, there are fees per wire transfer 1679 when a :term:`reserve` is :term:`closed <close>` 1680 and for :term:`aggregate` :term:`wire transfers <wire transfer>` 1681 to the :term:`merchant`. 1682 1683 fresh 1684 a :term:`coin` is fresh if its public key is only known to the customer 1685 1686 JSON 1687 JavaScript Object Notation (JSON) is a 1688 serialization format derived from the JavaScript language which is 1689 commonly used in the Taler protocol as the payload of HTTP requests 1690 and responses. 1691 1692 kappa 1693 security parameter used in the :term:`refresh` protocol. Defined to be 3. 1694 The probability of successfully evading the income transparency with the 1695 refresh protocol is 1:kappa. 1696 1697 libeufin 1698 Kotlin component that implements a regional currency bank and an 1699 adapter to communicate via EBICS with European core banking systems. 1700 1701 link 1702 specific step in the :term:`refresh` protocol that an exchange must offer 1703 to prevent abuse of the :term:`refresh` mechanism. The link step is 1704 not needed in normal operation, it just must be offered. 1705 1706 master key 1707 offline key used by the exchange to certify denomination keys and 1708 message signing keys 1709 1710 melt 1711 step of the :term:`refresh` protocol where a :term:`dirty` :term:`coin` 1712 is invalidated to be reborn :term:`fresh` in a subsequent 1713 :term:`reveal` step. 1714 1715 merchant 1716 party receiving payments (usually in return for goods or services) 1717 1718 message signing key 1719 key used by the exchange to sign online messages, other than coins 1720 1721 order 1722 offer made by the merchant to a wallet; pre-cursor to 1723 a contract where the wallet is not yet fixed. Turns 1724 into a :term:`contract` when a wallet claims the order. 1725 1726 owner 1727 a coin is owned by the entity that knows the private key of the coin 1728 1729 relative time 1730 method of keeping time in :term:`GNUnet` where the time is represented 1731 as a relative number of microseconds. Thus, a relative time specifies 1732 an offset or a duration, but not a date. Called relative time in 1733 contrast to :term:`absolute time`. 1734 1735 recoup 1736 Operation by which an exchange returns the value of coins affected 1737 by a :term:`revocation <revoke>` to their :term:`owner`, either by allowing the owner to 1738 withdraw new coins or wiring funds back to the bank account of the :term:`owner`. 1739 1740 planchet 1741 precursor data for a :term:`coin`. A planchet includes the coin's internal 1742 secrets (coin private key, blinding factor), but lacks the RSA signature 1743 of the :term:`exchange`. When :term:`withdrawing <withdraw>`, a :term:`wallet` 1744 creates and persists a planchet before asking the exchange to sign it to 1745 get the coin. 1746 1747 purchase 1748 Refers to the overall process of negotiating a :term:`contract` and then 1749 making a payment with :term:`coins <coin>` to a :term:`merchant`. 1750 1751 privacy policy 1752 Statement of an operator how they will protect the privacy of users. 1753 1754 proof 1755 Message that cryptographically demonstrates that a particular claim is correct. 1756 1757 proposal 1758 a list of :term:`contract terms` that has been completed and signed by the 1759 merchant backend. 1760 1761 refresh 1762 operation by which a :term:`dirty` :term:`coin` is converted into one or more 1763 :term:`fresh` coins. Involves :term:`melting <melt>` the :term:`dirty` coins and 1764 then :term:`revealing <reveal>` so-called :term:`transfer keys <transfer key>`. 1765 1766 refund 1767 operation by which a merchant steps back from the right to funds that he 1768 obtained from a :term:`deposit` operation, giving the right to the funds back 1769 to the customer 1770 1771 refund transaction id 1772 unique number by which a merchant identifies a :term:`refund`. Needed 1773 as refunds can be partial and thus there could be multiple refunds for 1774 the same :term:`purchase`. 1775 1776 reserve 1777 accounting mechanism used by the exchange to track customer funds 1778 from incoming :term:`wire transfers <wire transfer>`. A reserve is created whenever 1779 a customer wires money to the exchange using a well-formed public key 1780 in the subject. The exchange then allows the customer's :term:`wallet` 1781 to :term:`withdraw` up to the amount received in :term:`fresh` 1782 :term:`coins <coin>` from the reserve, thereby emptying the reserve. If a 1783 reserve is not emptied, the exchange will eventually :term:`close` it. 1784 1785 Other definition: Funds set aside for future use; either the balance of a customer at the 1786 exchange ready for withdrawal, or the funds kept in the exchange;s bank 1787 account to cover obligations from coins in circulation. 1788 1789 reveal 1790 step in the :term:`refresh` protocol where some of the transfer private 1791 keys are revealed to prove honest behavior on the part of the wallet. 1792 In the reveal step, the exchange returns the signed :term:`fresh` coins. 1793 1794 revoke 1795 exceptional operation by which an exchange withdraws a denomination from 1796 circulation, either because the signing key was compromised or because 1797 the exchange is going out of operation; unspent coins of a revoked 1798 denomination are subjected to recoup. 1799 1800 sharing 1801 users can share ownership of a :term:`coin` by sharing access to the coin's 1802 private key, thereby allowing all co-owners to spend the coin at any 1803 time. 1804 1805 spend 1806 operation by which a customer gives a merchant the right to deposit 1807 coins in return for merchandise 1808 1809 transfer key 1810 special cryptographic key used in the :term:`refresh` protocol, some of which 1811 are revealed during the :term:`reveal` step. Note that transfer keys have, 1812 despite the name, no relationship to :term:`wire transfers <wire transfer>`. They merely 1813 help to transfer the value from a :term:`dirty` coin to a :term:`fresh` coin 1814 1815 terms 1816 the general terms of service of an operator, possibly including 1817 the :term:`privacy policy`. Not to be confused with the 1818 :term:`contract terms` which are about the specific purchase. 1819 1820 transaction 1821 method by which ownership is exclusively transferred from one entity 1822 1823 user 1824 any individual using the Taler payment system 1825 (see :term:`customer`, :term:`buyer`, :term:`merchant`). 1826 1827 version 1828 Taler uses various forms of versioning. There is a database 1829 schema version (stored itself in the database, see \*-0000.sql) describing 1830 the state of the table structure in the database of an :term:`exchange`, 1831 :term:`auditor` or :term:`merchant`. There is a protocol 1832 version (CURRENT:REVISION:AGE, see GNU libtool) which specifies 1833 the network protocol spoken by an :term:`exchange` or :term:`merchant` 1834 including backwards-compatibility. And finally there is the software 1835 release version (MAJOR.MINOR.PATCH, see https://semver.org/) of 1836 the respective code base. 1837 1838 wallet 1839 software running on a customer's computer; withdraws, stores and 1840 spends coins 1841 1842 WebExtension 1843 Cross-browser API used to implement the GNU Taler wallet browser extension. 1844 1845 wire gateway 1846 API used by the exchange to talk with some real-time gross settlement system 1847 (core banking system, blockchain) to notice inbound credits wire transfers 1848 (during withdraw) and to trigger outbound debit wire transfers (primarily 1849 for deposits). 1850 1851 wire transfer 1852 a wire transfer is a method of sending funds between :term:`bank` accounts 1853 1854 wire transfer identifier 1855 Subject of a wire transfer from the exchange to a merchant; 1856 set by the aggregator to a random nonce which uniquely 1857 identifies the transfer. 1858 1859 withdraw 1860 operation by which a :term:`wallet` can convert funds from a :term:`reserve` to 1861 fresh coins 1862 1863 zombie 1864 :term:`coin` where the respective :term:`denomination key` is past its 1865 :term:`deposit` :term:`expiration <expired>` time, but which is still (again) valid 1866 for an operation because it was :term:`melted <melt>` while it was still 1867 valid, and then later again credited during a :term:`recoup` process 1868 1869 1870 1871 Developer Tools 1872 =============== 1873 1874 This section describes various internal programs to make life easier for the 1875 developer. 1876 1877 1878 taler-harness 1879 ------------- 1880 1881 **taler-harness deployment gen-coin-config** is a tool to simplify Taler configuration generation. 1882 1883 1884 **taler-harness deployment gen-coin-config** 1885 [**-min-amount**=**\ \ *VALUE*] 1886 [**-max-amount**=**\ \ *VALUE*]