taler-developer-manual.rst (65653B)
1 .. 2 This file is part of GNU TALER. 3 4 Copyright (C) 2014-2025 Taler Systems SA 5 6 TALER is free software; you can redistribute it and/or modify it under the 7 terms of the GNU Affero General Public License as published by the Free Software 8 Foundation; either version 3.0, or (at your option) any later version. 9 10 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 11 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 12 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 13 14 You should have received a copy of the GNU Affero General Public License along with 15 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 16 17 @author Christian Grothoff 18 19 General Developer Manual 20 ######################## 21 22 .. note:: 23 24 This manual contains information for developers working on GNU Taler 25 and related components. It is not intended for a general audience. 26 27 28 Project Overview 29 ================ 30 31 GNU Taler consists of a large (and growing) number of components 32 in various Git repositories. The following list gives a first 33 overview: 34 35 * exchange: core payment processing logic with a REST API, plus various 36 helper processes for interaction with banks and cryptographic 37 computations. Also includes the logic for the auditor and an 38 in-memory "bank" API implementation for testing. 39 40 * libeufin: implementation of the "bank" API using the EBICS protocol 41 used by banks in the EU. Allows an exchange to interact with 42 European banks. 43 44 * taler-magnet-bank: implementation of the "bank" API using the Magnet Bank 45 API. Allows an exchange to interact with Magnet Bank. 46 47 * taler-cyclos: implementation of the "bank" API using the Cyclos API. Allows an exchange to interact with a Cyclos network. 48 49 * taler-wise: implementation of the "bank" API using the Wise API. Allows an exchange to interact with Wise. 50 51 * depolymerization: implementation of the "bank" API on top of 52 blockchains, specifically Bitcoin and Ethereum. Allows an exchange 53 to interact with crypto-currencies. 54 55 * merchant: payment processing backend to be run by merchants, 56 offering a REST API. 57 58 * wallet-core: platform-independent implementation of a wallet to be run by 59 normal users. Includes also the WebExtension for various browsers. 60 Furthermore, includes various single-page apps used by other 61 components (especially as libeufin and merchant). Also includes 62 command-line wallet and tools for testing. 63 64 * taler-android: Android Apps including the Android wallet, the 65 Android point-of-sale App and the Android casher app. 66 67 * taler-ios: iOS wallet App. 68 69 * sync: backup service, provides a simple REST API to allow users to 70 make encrypted backups of their wallet state. 71 72 * anastasis: key escrow service, provides a simple REST API to allow 73 users to distribute encryption keys across multiple providers and 74 define authorization policies for key recovery. 75 76 * taler-mdb: integration of Taler with the multi-drop-bus (MDB) API 77 used by vending machines. Allows Taler payments to be integrated 78 with vending machines. 79 80 * gnu-taler-payment-for-woocommerce: payment plugin for the 81 woocommerce (wordpress) E-commerce solution. 82 83 * twister: man-in-the-middle proxy for tests that require fuzzing a 84 REST/JSON protocol. Used for some of our testing. 85 86 * challenger: implementation of an OAuth 2.0 provider that can be used 87 to verify that a user can receive SMS or E-mail at particular addresses. 88 Used as part of KYC processes of the exchange. 89 90 * taler-mailbox: messaging service used to store and forward payment 91 messages to Taler wallets. 92 93 * taldir: directory service used to lookup Taler wallet addresses for 94 sending invoices or payments to other wallets. 95 96 * taler-merchant-demos: various demonstration services operated at 97 'demo.taler.net', including a simple shop and a donation page. 98 99 There are other important repositories without code, including: 100 101 * gana: Hosted on git.gnunet.org, this repository defines various 102 constants used in the GNU Taler project. 103 104 * docs: documentation, including this very document. 105 106 * marketing: various presentations, papers and other resources for 107 outreach. 108 109 * large-media: very large data objects, such as videos. 110 111 * www: the taler.net website. 112 113 Fundamentals 114 ============ 115 116 Versioning 117 ---------- 118 119 A central rule is to never break anything for any dependency. To accomplish 120 this, we use versioning, of the APIs, database schema and the protocol. The 121 database versioning approach is described in the :ref:`Database schema 122 versioning <DatabaseVersioning>` section. Here, we will focus on API and 123 protocol versioning. 124 125 The key issue we need to solve with protocols and APIs (and that does not 126 apply to database versioning) is being able to introduce and remove features 127 without requiring a flag day where all components must update at the same 128 time. For this, we use GNU libtool style versioning with MAJOR:REVISION:AGE 129 and *not* semantic versioning (SEMVER). With GNU libtool style versioning, 130 first the REVISION should be increased on every change to the respective code. 131 Then, each time a feature is introduced or deprecated, the MAJOR and AGE 132 numbers are increased. Whenever an API is actually removed the AGE number is 133 reduced to match the distance since the removed API was deprecated. Thus, if 134 some client implements version X of the protocol (including not using any APIs 135 that have been deprecated), it is compatible for any implementation where 136 MAJOR is larger or equal to X, and MAJOR minus AGE is smaller or equal to X. 137 REVISION is not used for expected compatibility issues and merely serves to 138 uniquely identify each version (in combination with MAJOR). 139 140 To evolve any implementation, it is thus critical to first of all never 141 just break an existing API or endpoint. The only acceptable modifications 142 are to return additional information (being aware of binary compatibility!) 143 or to accept additional optional arguments (again, in a way that does not 144 break existing users). Thus, the most common way to introduce changes will 145 be the addition of new endpoints. Breaking existing endpoints is only ever 146 at best acceptable while in the process of introducing it and if you are 147 absolutely sure that there are zero users in other components. 148 149 When removing endpoints (or fields being returned), you must first deprecate 150 the existing API (incrementing MAJOR and AGE) and then wait for all clients, 151 including all clients in operation (e.g. Android and iOS Apps, e-commerce 152 integrations, etc.) to upgrade to a protocol implementation above the 153 deprecated MAJOR revision. Only then you should remove the endpoint and reduce 154 AGE. 155 156 To document these changes, please try to use ``@since`` annotations in the API 157 specifications to explain the MAJOR revision when a feature became available, 158 but most importantly use ``@deprecated X`` annotations to indicate that an API 159 was deprecated and will be removed once MAJOR minus AGE is above X. When using 160 an API, use the ``/config`` endpoints to check for compatibility and show a 161 warning if the version(s) you support and the version(s) offered by the server 162 are incompatible. 163 164 165 Tagging and Package Versioning 166 ------------------------------ 167 168 Release tags are of the form ``v${major}.${minor}.${patch}``. Release tags *should* be 169 annotated git tags. 170 171 We usually consider Debian packaging files (in ``debian/``) to be part of a release. 172 When only the Debian packaging files need to be changed, there are two options: 173 174 * Make a new patch release (``v${major}.${minor}.${patch+1}``) 175 * Make a Debian release: 176 177 * Debian version now includes a revision: ``${major}.${minor}.${patch}-${debrevision}`` 178 * The tag is Debian-specific: ``debian-${major}.${minor}.${patch}-${debrevision}`` 179 180 All source repos *should* include a ``contrib/bump`` script that automates bumping the\ 181 version in all relevant source and packaging files. 182 In the future, we might add an option to the script to only release a packaging bump. 183 Right now, that process is manual. 184 185 We support tagged and published pre-release versions via tags of the form ``v${major}.${minor}.${patch}-dev.${n}``. 186 The corresponding Debian version must be ``${major}.${minor}.${patch}~dev${n}``. 187 188 Nightly Debian packages should follow the `Debian conventions <https://wiki.debian.org/Versioning>`__ of ``{upcoming_version}~git{date}.{hash}-{revision}``. 189 190 Testing Tools 191 ------------- 192 193 For full ``make check`` support, install these programs: 194 195 - `jq <https://github.com/stedolan/jq>`__ 196 - `curl <http://curl.haxx.se>`__ 197 - `faketime <https://github.com/wolfcw/libfaketime>`__ 198 199 The ``make check`` should be able to function without them, but 200 their presence permits some tests to run that would otherwise be skipped. 201 202 Manual Testing Database Reset 203 ----------------------------- 204 205 Sometimes ``make check`` will fail with some kind of database (SQL) 206 error, perhaps with a message like ``OBJECT does not exist`` in the 207 ``test-suite.log`` file, where ``OBJECT`` is the name of a table or function. 208 In that case, it may be necessary to reset the ``talercheck`` database 209 with the commands: 210 211 .. code-block:: console 212 213 $ dropdb talercheck 214 $ createdb talercheck 215 216 This is because, at the moment, there is no support for 217 doing these steps automatically in the ``make check`` flow. 218 219 (If ``make check`` still fails after the reset, file a bug report as usual.) 220 221 Bug Tracking 222 ------------ 223 224 Bug tracking is done with Mantis (https://www.mantisbt.org/). The bug tracker 225 is available at `<https://bugs.taler.net>`_. A registration on the Web site is 226 needed in order to use the bug tracker, only read access is granted without a 227 login. 228 229 We use the following conventions for the bug states: 230 231 * NEW: Incoming bugs are in 'new' so that management (or developers) 232 can easily identify those that need to be checked (report correct? 233 something we want to fix?), prioritized and targeted for releases. 234 "NEW" bugs are never assigned to a developer. 235 236 * FEEDBACK: When blocked on feedback from reporter or other developer. 237 Assigned to other developer (but cannot be assigned to reporter, 238 in this case MAY remain associated with the developer who expects 239 the feedback). If a bug is on feedback, it automatically should be 240 considered to be high-priority to give the feedback (as it is 241 blocking someone else!). 242 243 * ACKNOWLEDGED: The bug has been reviewed, but no decision about 244 what action to take has been made yet. Should not be worked on 245 until management (or a developer) comes up with a plan. 246 "ACKNOWLEDGED" bugs should NOT be assigned to a developer. 247 248 * CONFIRMED: This is a real issue that should be worked on, but 249 is not yet actively worked on. If working on this bug requires 250 other bugs to be fixed first, they should be added as 251 child-bugs (via relationships). Developers are always welcome 252 to self-assign bugs that are "CONFIRMED" if they start to work 253 on a bug. "CONFIRMED" bugs should NOT be assigned to a developer. 254 255 * ASSIGNED: The specific developer the bug is assigned to is 256 **actively** working on the issue. Developers should strive to 257 not have more than 5-10 bugs assigned to them at any time. 258 Only having one assigned to you is totally OK! 259 Developers should aggressively un-assign bugs that they are 260 blocked on, cannot make progress on, or are no longer actively 261 working on (but of course, better *resolve* them before 262 moving on if possible). If the bug remains open, it probably 263 should go back to "CONFIRMED" or "ACKNOWLEDGED". 264 265 * RESOLVED: The bug has been fixed in Git. 266 267 * CLOSED: An official release was made with the fix in it. 268 269 When developers want to keep an eye on certain bugs, they should 270 **monitor** them. Multiple developers can be monitoring a bug, but 271 it can only be assigned to one. Developers should also keep an 272 eye on the roadmap (by release), bug categories they care about, 273 and of course priorities / severities. 274 275 We use **tags** to categorize bugs. Common tags that also imply 276 some urgency include (in alphabetical order): 277 278 * accounting: issues required for accounting (such as taxes by merchants) 279 * compliance: issues related to regulatory compliance 280 * $CUSTOMER: issues requested by a particular customer 281 * performance: performance problems or ideas for improvement 282 * security: security issues (including planned improvements to security) 283 * UX: user experience issues 284 285 These tags **should** be attached to "NEW" bugs if they apply. 286 287 288 Code Repositories 289 ----------------- 290 291 Taler code is versioned with Git. For those users without write access, all the 292 codebases are found at the following URL: 293 294 .. code-block:: none 295 296 git://git.taler.net/<repository> 297 298 A complete list of all the existing repositories is currently found at 299 `<https://git.taler.net/>`_. 300 301 302 Committing code 303 --------------- 304 305 Before you can obtain Git write access, you must sign the copyright 306 agreement. As we collaborate closely with GNUnet, we use their 307 copyright agreement -- with the understanding that your contributions 308 to GNU Taler are included in the assignment. You can find the 309 agreement on the `GNUnet site <https://gnunet.org/en/copyright.html>`_. 310 Please sign and mail it to Christian Grothoff as he currently collects 311 all the documents for GNUnet e.V. 312 313 To obtain Git access, you need to send us your SSH public key. Most core 314 team members have administrative Git access, so simply contact whoever 315 is your primary point of contact so far. You can 316 find instructions on how to generate an SSH key 317 in the `Git book <https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key>`_. 318 If you have been granted write access, you first of all must change the URL of 319 the respective repository to: 320 321 .. code-block:: none 322 323 ssh://git@git.taler.net/<repository> 324 325 For an existing checkout, this can be done by editing the ``.git/config`` file. 326 327 The server is configured to reject all commits that have not been signed with 328 GnuPG. If you do not yet have a GnuPG key, you must create one, as explained 329 in the `GNU Privacy Handbook <https://www.gnupg.org/gph/en/manual/c14.html>`_. 330 You do not need to share the respective public key with us to make commits. 331 However, we recommend that you upload it to key servers, put it on your 332 business card and personally meet with other GNU hackers to have it signed 333 such that others can verify your commits later. 334 335 To sign all commits, you should run 336 337 .. code-block:: console 338 339 $ git config --global commit.gpgsign true 340 341 You can also sign individual commits only by adding the ``-S`` option to the 342 ``git commit`` command. If you accidentally already made commits but forgot 343 to sign them, you can retroactively add signatures using: 344 345 .. code-block:: console 346 347 $ git rebase -S 348 349 350 Whether you commit to a personal branch (recommended: ``dev/$USER/...``), 351 a feature branch or to ``master`` should 352 depend on your level of comfort and the nature of the change. As a general 353 rule, the code in ``master`` must always build and tests should always pass, at 354 least on your own system. However, we all make mistakes and you should expect 355 to receive friendly reminders if your change did not live up to this simple 356 standard. We plan to move to a system where the CI guarantees this invariant 357 in the future. 358 359 In order to keep a linear and clean commits history, we advise to avoid 360 merge commits and instead always rebase your changes before pushing to 361 the ``master`` branch. If you commit and later find out that new commits were 362 pushed, the following command will pull the new commits and rebase yours 363 on top of them. 364 365 .. code-block:: console 366 367 # -S instructs Git to (re)sign your commits 368 $ git pull --rebase -S 369 370 371 372 Observing changes 373 ----------------- 374 375 Every commit to the ``master`` branch of any of our public repositories 376 (and almost all are public) is automatically sent to the 377 gnunet-svn@gnu.org mailinglist. That list is for Git commits only, 378 and must not be used for discussions. It also carries commits from 379 our main dependencies, namely GNUnet and GNU libmicrohttpd. While 380 it can be high volume, the lists is a good way to follow overall 381 development. 382 383 384 Code generator usage policy 385 --------------------------- 386 387 We do neither encourage nor discourage the use of tools for code generation. 388 It is up to the individual developer to decide if a tool is acceptable for 389 a particular task. But of course, we do encourage you to use FLOSS tools 390 and we MUST NOT become dependent on non-free software! That said, if you 391 use tools, you must document their use and in particular satisfy the 392 `NLnet policy on the use of "AI" <https://nlnet.nl/news/2025/20250829-policy-on-use-of-AI.html>`__. 393 394 Specifically, we ask developers to always put generated code into a *separate* 395 Git commit and to include the full prompt in the commit message. Naturally, 396 you may clean up the code generator's output, but then you should do so in 397 separate Git commits (and of course only merge into master/stable after the 398 clean up is complete). But do preserve (not squash!) the commit with the 399 generated code so that it remains documented what the prompts were and which 400 code is generated. This will go a long way to keep code auditors sane! 401 402 403 Communication 404 ------------- 405 406 For public discussions we use the taler@gnu.org mailinglist. All developers 407 should subscribe to the low-volume Taler mailinglist. There are separate 408 low-volume mailinglists for gnunet-developers (@gnu.org) and for libmicrohttpd 409 (@gnu.org). For internal discussions we use https://mattermost.taler.net/ 410 (invitation only, but also archived). 411 412 413 What to put in bootstrap 414 ------------------------ 415 416 Each repository has a ``bootstrap`` script, which contains commands for the 417 developer to run after a repository checkout (i.e., after ``git clone`` or 418 ``git pull``). 419 Typically, this updates and initializes submodules, prepares the tool chain, 420 and runs ``autoreconf``. 421 The last step generates the ``configure`` script, whether for immediate use or 422 for inclusion in the distribution tarball. 423 424 One common submodule is ``contrib/gana``, which pulls from the 425 `GNUnet GANA repository <https://git.gnunet.org/gana.git/>`__. 426 For example, in the 427 `Taler exchange repository <https://git.taler.net/exchange.git>`__, 428 the bootstrap script eventually runs the ``git submodule update --init`` command 429 early on, and later runs script ``./contrib/gana-generate.sh``, which 430 generates files such as ``src/include/taler_signatures.h``. 431 432 Thus, to update that file, you need to: 433 434 - (in GANA repo) Find a suitable (unused) name and number for the Signature 435 Purposes database. 436 437 - Add it to GANA, in ``gnunet-signatures/registry.rec``. 438 (You can check for uniqueness with the ``recfix`` utility.) 439 440 - Commit the change, and push it to the GANA Git repo. 441 442 - (in Taler Repo) Run the ``contrib/gana-latest.sh`` script. 443 444 - Bootstrap, configure, do ``make install``, ``make check``, etc. 445 (Basically, make sure the change does not break anything.) 446 447 - Commit the submodule change, and push it to the Taler exchange Git repo. 448 449 A similar procedure is required for other databases in GANA. 450 See file ``README`` in the various directories for specific instructions. 451 452 453 Debian and Ubuntu Repositories 454 ============================== 455 456 We package our software for Debian and Ubuntu. 457 458 Nightly Repositories 459 -------------------- 460 461 To try the latest, unstable and untested versions of packages, 462 you can add the nightly package sources. 463 464 .. code-block:: shell-session 465 466 # For Debian (trixie) 467 $ curl -sS https://deb.taler.net/apt-nightly/taler-trixie-ci.sources \ 468 | tee /etc/apt/sources.list.d/taler-trixie-nightly.sources 469 470 471 Taler Deployment on gv.taler.net 472 ================================ 473 474 This section describes the GNU Taler deployment on ``gv.taler.net``. ``gv`` 475 is our server at BFH. It hosts the Git repositories, Web sites, CI and other 476 services. Developers can receive an SSH account and e-mail alias for the 477 system, you should contact Javier, Christian or Florian. As with Git, ask 478 your primary team contact for shell access if you think you need it. 479 480 481 DNS 482 --- 483 484 DNS records for taler.net are controlled by the GNU Taler maintainers, 485 specifically Christian and Florian, and our system administrator, Javier. If 486 you need a sub-domain to be added, please contact one of them. 487 488 489 User Acccounts 490 -------------- 491 492 On ``gv.taler.net``, there are three system users that are set up to 493 serve Taler on the Internet: 494 495 - ``head``: serves ``*.head.taler.net`` and gets automatically 496 built by Buildbot every 2 hours from the ``sandcastle-ng.git``. 497 Master key may be reset occasionally 498 499 - ``taler-test``: serves ``*.test.taler.net`` and does *NOT* get 500 automatically built, and runs more recent tags and/or unreleased 501 versions of Taler components. Master key may be reset 502 occasionally. 503 504 - ``demo``: serves ``*.demo.taler.net``. Never automatically built. 505 Master key is retained. 506 507 Demo Upgrade Procedure 508 ====================== 509 510 #. Login as the ``demo`` user on ``gv.taler.net``. 511 #. Pull the latest ``sandcastle-ng.git`` code in checkout at ``$HOME/sandcastle-ng``. 512 #. Run ``systemctl --user restart container-taler-sandcastle-demo.service`` 513 #. Refer to the sandcastle-ng README (https://git.taler.net/sandcastle-ng.git/about/) 514 for more info. 515 516 517 Upgrading the ``demo`` environment should be done with care, and ideally be 518 coordinated on the mailing list before. It is our goal for ``demo`` to always 519 run a "working version" that is compatible with various published wallets. 520 Please use the :doc:`demo upgrade checklist <../checklists/checklist-demo-upgrade>` to make 521 sure everything is working. 522 Nginx is already configured to reach the services as exported by the user unit. 523 524 525 Tagging components 526 ------------------ 527 528 All Taler components must be tagged with git before they are deployed on the 529 ``demo`` environment, using a tag of the following form: 530 531 .. code-block:: none 532 533 demo-YYYY-MM-DD-SS 534 YYYY = year 535 MM = month 536 DD = day 537 SS = serial 538 539 Environments and Builders on taler.net 540 ====================================== 541 542 Buildbot implementation 543 ----------------------- 544 545 GNU Taler uses a buildbot implementation (front end at https://buildbot.taler.net) to manage continuous integration. Buildbot documentation is at https://docs.buildbot.net/. 546 547 Here are some highlights: 548 549 - The WORKER is the config that that lives on a shell account on a localhost (taler.net), where this host has buildbot-worker installed. The WORKER executes the commands that perform all end-functions of buildbot. 550 551 - The WORKER running buildbot-worker receives these commands by authenticating and communicating with the buildbot server using parameters that were specified when the worker was created in that shell account with the ``buildbot-worker`` command. 552 553 - The buildbot server's master.cfg file contains FACTORY declarations which specify the commands that the WORKER will run on localhost. 554 555 - The FACTORY is tied to the WORKER in master.cfg by a BUILDER. 556 557 - The master.cfg also allows for SCHEDULER that defines how and when the BUILDER is executed. 558 559 - Our master.cfg file is checked into git, and then periodically updated on a particular account on taler.net (ask Christian for access if needed). Do not edit this file directly/locally on taler.net, but check changes into Git. 560 561 562 Best Practices: 563 564 - When creating a new WORKER in the ``master.cfg`` file, leave a comment specifying the server and user account that this WORKER is called from. (At this time, taler.net is the only server used by this implementation, but it's still good practice.) 565 566 - Create a worker from a shell account with this command: ``buildbot-worker create-worker <workername> localhost <username> <password>`` 567 568 Then make sure there is a WORKER defined in master.cfg like: ``worker.Worker("<username>", "<password>")`` 569 570 Test builder 571 ------------ 572 573 This builder (``test-builder``) compiles and starts every Taler component. 574 The associated worker is run by the ``taler-test`` Gv user, via the SystemD 575 unit ``buildbot-worker-taler``. The following commands start/stop/restart 576 the worker: 577 578 .. code-block:: 579 580 systemctl --user start buildbot-worker-taler 581 systemctl --user stop buildbot-worker-taler 582 systemctl --user restart buildbot-worker-taler 583 584 .. note:: 585 the mentioned unit file can be found at ``deployment.git/systemd-services/`` 586 587 Wallet builder 588 -------------- 589 590 This builder (``wallet-builder``) compiles every Taler component 591 and runs the wallet integration tests. The associated worker is 592 run by the ``walletbuilder`` Gv user, via the SystemD unit ``buildbot-worker-wallet``. 593 The following commands start/stop/restart the worker: 594 595 .. code-block:: 596 597 systemctl --user start buildbot-worker-wallet 598 systemctl --user stop buildbot-worker-wallet 599 systemctl --user restart buildbot-worker-wallet 600 601 .. note:: 602 the mentioned unit file can be found at ``deployment.git/systemd-services/`` 603 604 Documentation Builder 605 --------------------- 606 607 All the Taler documentation is built by the user ``docbuilder`` that 608 runs a Buildbot worker. The following commands set the ``docbuilder`` up, 609 starting with an empty home directory. 610 611 .. code-block:: console 612 613 # Log-in as the 'docbuilder' user. 614 615 $ cd $HOME 616 $ git clone git://git.taler.net/deployment 617 $ ./deployment/bootstrap-docbuilder 618 619 # If the previous step worked, the setup is 620 # complete and the Buildbot worker can be started. 621 622 $ buildbot-worker start worker/ 623 624 625 Website Builder 626 --------------- 627 628 629 Taler Websites, ``www.taler.net`` and ``stage.taler.net``, are built by the 630 user ``taler-websites`` by the means of a Buildbot worker. The following 631 commands set the ``taler-websites`` up, starting with an empty home directory. 632 633 .. code-block:: console 634 635 # Log-in as the 'taler-websites' user. 636 637 $ cd $HOME 638 $ git clone git://git.taler.net/deployment 639 $ ./deployment/bootstrap-sitesbuilder 640 641 # If the previous step worked, the setup is 642 # complete and the Buildbot worker can be started. 643 644 $ buildbot-worker start worker/ 645 646 647 Code coverage 648 ------------- 649 650 Code coverage tests are run by the ``lcovworker`` user, and are also driven 651 by Buildbot. 652 653 .. code-block:: console 654 655 # Log-in as the 'lcovworker' user. 656 657 $ cd $HOME 658 $ git clone git://git.taler.net/deployment 659 $ ./deployment/bootstrap-taler lcov 660 661 # If the previous step worked, the setup is 662 # complete and the Buildbot worker can be started. 663 664 $ buildbot-worker start worker/ 665 666 The results are then published at ``https://lcov.taler.net/``. 667 668 Producing auditor reports 669 ------------------------- 670 671 Both 'test' and 'demo' setups get their auditor reports compiled 672 by a Buildbot worker. The following steps get the reports compiler 673 prepared. 674 675 .. code-block:: console 676 677 # Log-in as <env>-auditor, with <env> being either 'test' or 'demo' 678 679 $ git clone git://git.taler.net/deployment 680 $ ./deployment/buildbot/bootstrap-scripts/prepare-auditorreporter <env> 681 682 # If the previous steps worked, then it should suffice to start 683 # the worker, with: 684 685 $ buildbot-worker start worker/ 686 687 688 .. _DatabaseVersioning: 689 690 Database schema versioning 691 -------------------------- 692 693 The PostgreSQL databases of the exchange and the auditor are versioned. 694 See the ``versioning.sql`` file in the respective directory for documentation. 695 696 Every set of changes to the database schema must be stored in a new 697 versioned SQL script. The scripts must have contiguous numbers. After 698 any release (or version being deployed to a production or staging 699 environment), existing scripts MUST be immutable. 700 701 Developers and operators MUST NOT make changes to database schema 702 outside of this versioning. All tables of a GNU Taler component should live in their own schema. 703 704 705 QA Plans 706 ======== 707 708 .. include:: ../checklists/qa-1.0.rst 709 710 711 Releases 712 ======== 713 714 .. include:: ../checklists/checklist-release.rst 715 716 Release Process 717 --------------- 718 719 This document describes the process for releasing a new version of the 720 various Taler components to the official GNU mirrors. 721 722 The following components are published on the GNU mirrors 723 724 - taler-exchange (exchange.git) 725 - taler-merchant (merchant.git) 726 - sync (sync.git) 727 - taler-mdb (taler-mdb.git) 728 - libeufin (libeufin.git) 729 - challenger (challenger.git) 730 - wallet-core (wallet-core.git) 731 732 Tagging 733 ------- 734 735 Tag releases with an **annotated** commit, like 736 737 .. code-block:: console 738 739 $ git tag -a v0.1.0 -m "Official release v0.1.0" 740 $ git push origin v0.1.0 741 742 743 Database for tests 744 ------------------ 745 746 For tests in the exchange and merchant to run, make sure that a database 747 *talercheck* is accessible by *$USER*. Otherwise tests involving the 748 database logic are skipped. 749 750 .. include:: ../frags/db-stores-sensitive-data.rst 751 752 Exchange, merchant 753 ------------------ 754 755 Set the version in ``configure.ac``. The commit being tagged should be 756 the change of the version. 757 758 Tag the current GANA version that works with the exchange and merchant and 759 checkout that tag of gana.git (instead of master). Otherwise, if there are 760 incompatible changes in GANA (like removed symbols), old builds could break. 761 762 Update the Texinfo documentation using the files from docs.git: 763 764 .. code-block:: console 765 766 # Get the latest documentation repository 767 $ cd $GIT/docs 768 $ git pull 769 $ make texinfo 770 # The *.texi files are now in _build/texinfo 771 # 772 # This checks out the prebuilt branch in the prebuilt directory 773 $ git worktree add prebuilt prebuilt 774 $ cd prebuilt 775 # Copy the pre-built documentation into the prebuilt directory 776 $ cp -r ../_build/texinfo . 777 # Push and commit to branch 778 $ git commit -a -S -m "updating texinfo" 779 $ git status 780 # Verify that all files that should be tracked are tracked, 781 # new files will have to be added to the Makefile.am in 782 # exchange.git as well! 783 $ git push 784 # Remember $REVISION of commit 785 # 786 # Go to exchange 787 $ cd $GIT/exchange/doc/prebuilt 788 # Update submodule to point to latest commit 789 $ git checkout $REVISION 790 791 Finally, the Automake ``Makefile.am`` files may have to be adjusted to 792 include new ``*.texi`` files or images. 793 794 For bootstrap, you will need to install 795 `GNU Recutils <https://www.gnu.org/software/recutils/>`_. 796 797 For the exchange test cases to pass, ``make install`` must be run first. 798 Without it, test cases will fail because plugins can't be located. 799 800 .. code-block:: console 801 802 $ ./bootstrap 803 $ ./configure # add required options for your system 804 $ make dist 805 $ tar -xf taler-$COMPONENT-$VERSION.tar.gz 806 $ cd taler-$COMPONENT-$VERSION 807 $ make install check 808 809 Wallet WebExtension 810 ------------------- 811 812 The version of the wallet is in *manifest.json*. The ``version_name`` 813 should be adjusted, and *version* should be increased independently on 814 every upload to the WebStore. 815 816 .. code-block:: console 817 818 $ ./configure 819 $ make dist 820 821 Upload to GNU mirrors 822 --------------------- 823 824 See https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads 825 826 Directive file: 827 828 .. code-block:: none 829 830 version: 1.2 831 directory: taler 832 filename: taler-exchange-0.1.0.tar.gz 833 symlink: taler-exchange-0.1.0.tar.gz taler-exchange-latest.tar.gz 834 835 Upload the files in **binary mode** to the ftp servers. 836 837 838 Creating Debian packages 839 ------------------------ 840 841 Our general setup is based on 842 https://wiki.debian.org/DebianRepository/SetupWithReprepro 843 844 First, update at least the version of the Debian package in 845 debian/changelog, and then run: 846 847 .. code-block:: bash 848 849 $ dpkg-buildpackage -rfakeroot -b -uc -us 850 851 in the respective source directory (GNUnet, exchange, merchant) to create the 852 ``.deb`` files. Note that they will be created in the parent directory. This 853 can be done on gv.taler.net, or on another (secure) machine. 854 Actual release builds should be done via the Docker images 855 that can be found in ``deployment.git`` under packaging. 856 857 On ``gv``, we use the ``aptbuilder`` user to manage the reprepro repository. 858 859 Next, the ``*.deb`` files should be copied to gv.taler.net, say to 860 ``/home/aptbuilder/incoming``. Then, run 861 862 .. code-block:: bash 863 864 # cd /home/aptbuilder/apt 865 # reprepro includedeb bullseye ~/incoming/*.deb 866 867 to import all Debian files from ``~/incoming/`` into the ``bullseye`` 868 distribution. If Debian packages were build against other distributions, 869 reprepro may need to be first configured for those and the import command 870 updated accordingly. 871 872 Finally, make sure to clean up ``~/incoming/`` (by deleting the 873 now imported ``*.deb`` files). 874 875 876 877 Continuous integration 878 ====================== 879 880 CI is done with Buildbot (https://buildbot.net/), and builds are 881 triggered by the means of Git hooks. The results are published at 882 https://buildbot.taler.net/ . 883 884 In order to avoid downtimes, CI uses a "blue/green" deployment 885 technique. In detail, there are two users building code on the system, 886 the "green" and the "blue" user; and at any given time, one is running 887 Taler services and the other one is either building the code or waiting 888 for that. 889 890 There is also the possibility to trigger builds manually, but this is 891 only reserved to "admin" users. 892 893 894 Internationalisation 895 ==================== 896 897 Internationalisation (a.k.a "translation") is handled using text-based 898 localization files named PO (Portable Object) holding pairs of original and 899 translated strings. 900 901 .. include:: dictionary.rst 902 903 904 iOS Apps 905 ======== 906 907 .. _Build-iOS-from-source: 908 909 Building Taler Wallet for iOS from source 910 ----------------------------------------- 911 912 The GNU Taler Wallet iOS app is in 913 `the official Git repository <https://git.taler.net/taler-ios.git>`__. 914 915 Compatibility 916 ^^^^^^^^^^^^^ 917 918 The minimum version of iOS supported is 15.0. 919 This app runs on all iPhone models at least as new as the iPhone 6S. 920 921 922 Building 923 ^^^^^^^^ 924 925 Before building the iOS wallet, you must first checkout the 926 `quickjs-tart repo <https://git.taler.net/quickjs-tart.git>`__ 927 and the 928 `wallet-core repo <https://git.taler.net/wallet-core.git>`__. 929 930 Have all 3 local repos (wallet-core, quickjs-tart, and this one) adjacent at 931 the same level (e.g. in a "GNU_Taler" folder) 932 Taler.xcworkspace expects the QuickJS framework sub-project to be at 933 ``../quickjs-tart/QuickJS-rt.xcodeproj``. 934 935 Build wallet-core first: 936 937 .. code-block:: shell-session 938 939 $ cd wallet-core 940 $ make embedded 941 $ open packages/taler-wallet-embedded/dist 942 943 then drag or move its product "taler-wallet-core-qjs.mjs" 944 into your quickjs-tart folder right at the top level. 945 946 Open Taler.xcworkspace, and set scheme / target to Taler_Wallet. Build&run... 947 948 Don't open QuickJS-rt.xcodeproj or TalerWallet.xcodeproj and build anything 949 there - all needed libraries and frameworks will be built automatically from 950 Taler.xcworkspace. 951 952 953 Android Apps 954 ============ 955 956 Android App Nightly Builds 957 -------------------------- 958 959 There are currently three Android apps in 960 `the official Git repository <https://git.taler.net/taler-android.git>`__: 961 962 * Wallet 963 [`CI <https://git.taler.net/taler-android.git/tree/wallet/.gitlab-ci.yml>`__] 964 * Merchant PoS Terminal 965 [`CI <https://git.taler.net/taler-android.git/tree/merchant-terminal/.gitlab-ci.yml>`__] 966 * Cashier 967 [`CI <https://git.taler.net/taler-android.git/tree/cashier/.gitlab-ci.yml>`__] 968 969 Their git repositories are `mirrored at Gitlab <https://gitlab.com/gnu-taler/taler-android>`__ 970 to utilize their CI 971 and `F-Droid <https://f-droid.org>`_'s Gitlab integration 972 to `publish automatic nightly builds <https://f-droid.org/docs/Publishing_Nightly_Builds/>`_ 973 for each change on the ``master`` branch. 974 975 All three apps publish their builds to the same F-Droid nightly repository 976 (which is stored as a git repository): 977 https://gitlab.com/gnu-taler/fdroid-repo-nightly 978 979 You can download the APK files directly from that repository 980 or add it to the F-Droid app for automatic updates 981 by clicking the following link (on the phone that has F-Droid installed). 982 983 `GNU Taler Nightly F-Droid Repository <fdroidrepos://gnu-taler.gitlab.io/fdroid-repo-nightly/fdroid/repo?fingerprint=55F8A24F97FAB7B0960016AF393B7E57E7A0B13C2D2D36BAC50E1205923A7843>`_ 984 985 .. note:: 986 Nightly apps can be installed alongside official releases 987 and thus are meant **only for testing purposes**. 988 Use at your own risk! 989 990 .. _Build-apps-from-source: 991 992 Building apps from source 993 ------------------------- 994 995 Note that this guide is different from other guides for building Android apps, 996 because it does not require you to run non-free software. 997 It uses the Merchant PoS Terminal as an example, but works as well for the other apps 998 if you replace ``merchant-terminal`` with ``wallet`` or ``cashier``. 999 1000 First, ensure that you have the required dependencies installed: 1001 1002 * Java Development Kit 8 or higher (default-jdk-headless) 1003 * git 1004 * unzip 1005 1006 Then you can get the app's source code using git: 1007 1008 .. code-block:: console 1009 1010 # Start by cloning the Android git repository 1011 $ git clone https://git.taler.net/taler-android.git 1012 1013 # Change into the directory of the cloned repository 1014 $ cd taler-android 1015 1016 # Find out which Android SDK version you will need 1017 $ grep -i compileSdkVersion merchant-terminal/build.gradle 1018 1019 The last command will return something like ``compileSdkVersion 29``. 1020 So visit the `Android Rebuilds <http://android-rebuilds.beuc.net/>`_ project 1021 and look for that version of the Android SDK there. 1022 If the SDK version is not yet available as a free rebuild, 1023 you can try to lower the ``compileSdkVersion`` in the app's ``merchant-terminal/build.gradle`` file. 1024 Note that this might break things 1025 or require you to also lower other versions such as ``targetSdkVersion``. 1026 1027 In our example, the version is ``29`` which is available, 1028 so download the "SDK Platform" package of "Android 10.0.0 (API 29)" 1029 and unpack it: 1030 1031 .. code-block:: console 1032 1033 # Change into the directory that contains your downloaded SDK 1034 $ cd $HOME 1035 1036 # Unpack/extract the Android SDK 1037 $ unzip android-sdk_eng.10.0.0_r14_linux-x86.zip 1038 1039 # Tell the build system where to find the SDK 1040 $ export ANDROID_SDK_ROOT="$HOME/android-sdk_eng.10.0.0_r14_linux-x86" 1041 1042 # Change into the directory of the cloned repository 1043 $ cd taler-android 1044 1045 # Build the merchant-terminal app 1046 $ ./gradlew :merchant-terminal:assembleRelease 1047 1048 If you get an error message complaining about build-tools 1049 1050 > Failed to install the following Android SDK packages as some licences have not been accepted. 1051 build-tools;29.0.3 Android SDK Build-Tools 29.0.3 1052 1053 you can try changing the ``buildToolsVersion`` in the app's ``merchant-terminal/build.gradle`` file 1054 to the latest "Android SDK build tools" version supported by the Android Rebuilds project. 1055 1056 After the build finished successfully, 1057 you will find your APK in ``merchant-terminal/build/outputs/apk/release/``. 1058 1059 Update translations 1060 ------------------- 1061 1062 Translations are managed with Taler's weblate instance: 1063 https://weblate.taler.net/projects/gnu-taler/ 1064 1065 To update translations, enter the taler-android git repository 1066 and ensure that the weblate remote exists: 1067 1068 .. code-block:: console 1069 1070 $ git config -l | grep weblate 1071 1072 If it does not yet exist (empty output), you can add it like this: 1073 1074 .. code-block:: console 1075 1076 $ git remote add weblate https://weblate.taler.net/git/gnu-taler/wallet-android/ 1077 1078 Then you can merge in translations commit from the weblate remote: 1079 1080 .. code-block:: console 1081 1082 # ensure you have latest version 1083 $ git fetch weblate 1084 1085 # merge in translation commits 1086 $ git merge weblate/master 1087 1088 Afterwards, build the entire project from source and test the UI 1089 to ensure that no erroneous translations (missing placeholders) are breaking things. 1090 1091 Release process 1092 --------------- 1093 1094 After extensive testing, the code making up a new release should get a signed git tag. 1095 The current tag format is: 1096 1097 * cashier-$VERSION 1098 * pos-$VERSION 1099 * wallet-$VERSION (where $VERSION has a v prefix) 1100 1101 .. code-block:: console 1102 1103 $ git tag -s $APP-$VERSION 1104 1105 F-Droid 1106 ^^^^^^^ 1107 Nightly builds get published automatically (see above) after pushing code to the official repo. 1108 Actual releases get picked up by F-Droid's official repository via git tags. 1109 So ensure that all releases get tagged properly. 1110 1111 Some information for F-Droid official repository debugging: 1112 1113 * Wallet: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.wallet.fdroid.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.wallet.fdroid/lastbuild>`__] 1114 * Cashier: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.cashier.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.cashier/lastbuild>`__] 1115 * PoS: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.merchantpos.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.merchantpos/lastbuild>`__] 1116 1117 Google Play 1118 ^^^^^^^^^^^ 1119 Google Play uploads are managed via `Fastlane <https://docs.fastlane.tools/getting-started/android/setup/>`__. 1120 Before proceeding, ensure that this is properly set up 1121 and that you have access to the Google Play API. 1122 1123 It is important to have access to the signing keys and Google Play access keys 1124 (JSON) and to ensure that the following environment variables are set 1125 correctly and made available to Fastlane: 1126 1127 .. code-block:: bash 1128 1129 TALER_KEYSTORE_PATH= 1130 TALER_KEYSTORE_PASS= 1131 TALER_KEYSTORE_WALLET_ALIAS= 1132 TALER_KEYSTORE_WALLET_PASS= 1133 TALER_KEYSTORE_POS_ALIAS= 1134 TALER_KEYSTORE_POS_PASS= 1135 TALER_KEYSTORE_CASHIER_ALIAS= 1136 TALER_KEYSTORE_CASHIER_PASS= 1137 TALER_JSON_KEY_FILE= 1138 1139 To release an app, enter into its respective folder and run fastlane: 1140 1141 .. code-block:: console 1142 1143 $ bundle exec fastlane 1144 1145 Then select the deploy option. 1146 1147 All uploads are going to the beta track by default. These can be promoted to 1148 production later or immediately after upload if you feel daring. It is also 1149 important to bump the version and build code with every release. 1150 1151 .. _Code-coverage: 1152 1153 Code Coverage 1154 ============= 1155 1156 Code coverage is done with the Gcov / Lcov 1157 (http://ltp.sourceforge.net/coverage/lcov.php) combo, and it is run 1158 nightly (once a day) by a Buildbot worker. The coverage results are 1159 then published at https://lcov.taler.net/ . 1160 1161 1162 Coding Conventions 1163 ================== 1164 1165 GNU Taler is developed primarily in C, Kotlin, Python, Swift and TypeScript. 1166 1167 Components written in C 1168 ----------------------- 1169 1170 These are the general coding style rules for Taler. 1171 1172 * Baseline rules are to follow GNU guidelines, modified or extended 1173 by the GNUnet style: https://docs.gnunet.org/handbook/gnunet.html#Coding-style 1174 1175 Naming conventions 1176 ^^^^^^^^^^^^^^^^^^ 1177 1178 * include files (very similar to GNUnet): 1179 1180 * if installed, must start with "``taler_``" (exception: platform.h), 1181 and MUST live in src/include/ 1182 * if NOT installed, must NOT start with "``taler_``" and 1183 MUST NOT live in src/include/ and 1184 SHOULD NOT be included from outside of their own directory 1185 * end in "_lib" for "simple" libraries 1186 * end in "_plugin" for plugins 1187 * end in "_service" for libraries accessing a service, i.e. the exchange 1188 1189 * binaries: 1190 1191 * taler-exchange-xxx: exchange programs 1192 * taler-merchant-xxx: merchant programs (demos) 1193 * taler-wallet-xxx: wallet programs 1194 * plugins should be libtaler_plugin_xxx_yyy.so: plugin yyy for API xxx 1195 * libtalerxxx: library for API xxx 1196 1197 * logging 1198 1199 * tools use their full name in GNUNET_log_setup 1200 (i.e. 'taler-exchange-offline') and log using plain 'GNUNET_log'. 1201 * pure libraries (without associated service) use 'GNUNET_log_from' 1202 with the component set to their library name (without lib or '.so'), 1203 which should also be their directory name (i.e. 'util') 1204 * plugin libraries (without associated service) use 'GNUNET_log_from' 1205 with the component set to their type and plugin name (without lib or '.so'), 1206 which should also be their directory name (i.e. 'exchangedb-postgres') 1207 * libraries with associated service) use 'GNUNET_log_from' 1208 with the name of the service, which should also be their 1209 directory name (i.e. 'exchange') 1210 * for tools with ``-l LOGFILE``, its absence means write logs to stderr 1211 1212 * configuration 1213 1214 * same rules as for GNUnet 1215 1216 * exported symbols 1217 1218 * must start with TALER_[SUBSYSTEMNAME]_ where SUBSYSTEMNAME 1219 MUST match the subdirectory of src/ in which the symbol is defined 1220 * from libtalerutil start just with ``TALER_``, without subsystemname 1221 * if scope is ONE binary and symbols are not in a shared library, 1222 use binary-specific prefix (such as TMH = taler-exchange-httpd) for 1223 globals, possibly followed by the subsystem (TMH_DB_xxx). 1224 1225 * structs: 1226 1227 * structs that are 'packed' and do not contain pointers and are 1228 thus suitable for hashing or similar operations are distinguished 1229 by adding a "P" at the end of the name. (NEW) Note that this 1230 convention does not hold for the GNUnet-structs (yet). 1231 * structs that are used with a purpose for signatures, additionally 1232 get an "S" at the end of the name. 1233 1234 * private (library-internal) symbols (including structs and macros) 1235 1236 * must not start with ``TALER_`` or any other prefix 1237 1238 * testcases 1239 1240 * must be called "test_module-under-test_case-description.c" 1241 1242 * performance tests 1243 1244 * must be called "perf_module-under-test_case-description.c" 1245 1246 Shell Scripts 1247 ------------- 1248 1249 Shell scripts should be avoided if at all possible. The only permissible uses of shell scripts 1250 in GNU Taler are: 1251 1252 * Trivial invocation of other commands. 1253 * Scripts for compatibility (e.g. ``./configure``) that must run on 1254 as many systems as possible. 1255 1256 When shell scripts are used, they ``MUST`` begin with the following ``set`` command: 1257 1258 .. code-block:: console 1259 1260 # Make the shell fail on undefined variables and 1261 # commands with non-zero exit status. 1262 $ set -eu 1263 1264 Kotlin 1265 ------ 1266 1267 We so far have no specific guidelines, please follow best practices 1268 for the language. 1269 1270 1271 Python 1272 ------ 1273 1274 Supported Python Versions 1275 ^^^^^^^^^^^^^^^^^^^^^^^^^ 1276 1277 Python code should be written and built against version 3.7 of Python. 1278 1279 Style 1280 ^^^^^ 1281 1282 We use `yapf <https://github.com/google/yapf>`_ to reformat the 1283 code to conform to our style instructions. 1284 A reusable yapf style file can be found in ``build-common``, 1285 which is intended to be used as a git submodule. 1286 1287 Python for Scripting 1288 ^^^^^^^^^^^^^^^^^^^^ 1289 1290 When using Python for writing small utilities, the following libraries 1291 are useful: 1292 1293 * ``click`` for argument parsing (should be preferred over argparse) 1294 * ``pathlib`` for path manipulation (part of the standard library) 1295 * ``subprocess`` for "shelling out" to other programs. Prefer ``subprocess.run`` 1296 over the older APIs. 1297 1298 1299 Swift 1300 ----- 1301 1302 Please follow best practices for the language. 1303 1304 1305 TypeScript 1306 ---------- 1307 1308 Please follow best practices for the language. 1309 1310 1311 Testing library 1312 =============== 1313 1314 This chapter is a VERY ABSTRACT description of how testing is 1315 implemented in Taler, and in NO WAY wants to substitute the reading of 1316 the actual source code by the user. 1317 1318 In Taler, a test case is an array of ``struct TALER_TESTING_Command``, 1319 informally referred to as ``CMD``, that is iteratively executed by the 1320 testing interpreter. This latter is transparently initiated by the 1321 testing library. 1322 1323 However, the developer does not have to define CMDs manually, but 1324 rather call the proper constructor provided by the library. For example, 1325 if a CMD is supposed to test feature ``x``, then the library would 1326 provide the ``TALER_TESTING_cmd_x ()`` constructor for it. Obviously, 1327 each constructor has its own particular arguments that make sense to 1328 test ``x``, and all constructors are thoroughly commented within the 1329 source code. 1330 1331 Internally, each CMD has two methods: ``run ()`` and ``cleanup ()``. The 1332 former contains the main logic to test feature ``x``, whereas the latter 1333 cleans the memory up after execution. 1334 1335 In a test life, each CMD needs some internal state, made by values it 1336 keeps in memory. Often, the test has to *share* those values with other 1337 CMDs: for example, CMD1 may create some key material and CMD2 needs this 1338 key material to encrypt data. 1339 1340 The offering of internal values from CMD1 to CMD2 is made by *traits*. A 1341 trait is a ``struct TALER_TESTING_Trait``, and each CMD contains an array 1342 of traits, that it offers via the public trait interface to other 1343 commands. The definition and filling of such array happens transparently 1344 to the test developer. 1345 1346 For example, the following example shows how CMD2 takes an amount object 1347 offered by CMD1 via the trait interface. 1348 1349 Note: the main interpreter and the most part of CMDs and traits are 1350 hosted inside the exchange codebase, but nothing prevents the developer 1351 from implementing new CMDs and traits within other codebases. 1352 1353 .. code-block:: c 1354 1355 /* Without loss of generality, let's consider the 1356 * following logic to exist inside the run() method of CMD1 */ 1357 ... 1358 1359 struct TALER_Amount *a; 1360 /** 1361 * the second argument (0) points to the first amount object offered, 1362 * in case multiple are available. 1363 */ 1364 if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a)) 1365 return GNUNET_SYSERR; 1366 ... 1367 1368 use(a); /* 'a' points straight into the internal state of CMD2 */ 1369 1370 In the Taler realm, there is also the possibility to alter the behaviour 1371 of supposedly well-behaved components. This is needed when, for example, 1372 we want the exchange to return some corrupted signature in order to 1373 check if the merchant backend detects it. 1374 1375 This alteration is accomplished by another service called *twister*. The 1376 twister acts as a proxy between service A and B, and can be programmed 1377 to tamper with the data exchanged by A and B. 1378 1379 Please refer to the Twister codebase (under the ``test`` directory) in 1380 order to see how to configure it. 1381 1382 1383 User-Facing Terminology 1384 ======================= 1385 1386 This section contains terminology that should be used and that should not be 1387 used in the user interface and help materials. 1388 1389 Terms to Avoid 1390 -------------- 1391 1392 Refreshing 1393 Refreshing is the internal technical terminology for the protocol to 1394 give change for partially spent coins 1395 1396 **Use instead**: "Obtaining change" 1397 1398 Charge 1399 Charge has two opposite meanings (charge to a credit card vs. charge a battery). 1400 This can confuse users. 1401 1402 **Use instead**: "Obtain", "Credit", "Debit", "Withdraw", "Top up" 1403 1404 Coin 1405 Coins are an internal construct, the user should never be aware that their balance 1406 is represented by coins of different denominations. 1407 1408 **Use instead**: "(Digital) Cash" or "(Wallet) Balance" 1409 1410 Consumer 1411 Has bad connotation of consumption. 1412 1413 **Use instead**: Customer or user. 1414 1415 Proposal 1416 The term used to describe the process of the merchant facilitating the download 1417 of the signed contract terms for an order. 1418 1419 **Avoid**. Generally events that relate to proposal downloads 1420 should not be shown to normal users, only developers. Instead, use 1421 "communication with merchant failed" if a proposed order can't be downloaded. 1422 1423 Anonymous E-Cash 1424 Should be generally avoided, since Taler is only anonymous for 1425 the customer. Also some people are scared of anonymity (which as 1426 a term is also way too absolute, as anonymity is hardly ever perfect). 1427 1428 **Use instead**: "Privacy-preserving", "Privacy-friendly" 1429 1430 Payment Replay 1431 The process of proving to the merchant that the customer is entitled 1432 to view a digital product again, as they already paid for it. 1433 1434 **Use instead**: In the event history, "re-activated digital content purchase" 1435 could be used. (FIXME: this is still not nice.) 1436 1437 Session ID 1438 See Payment Replay. 1439 1440 Order 1441 Too ambiguous in the wallet. 1442 1443 **Use instead**: Purchase 1444 1445 Fulfillment URL 1446 URL that serves the digital content that the user purchased 1447 with their payment. Can also be something like a donation receipt. 1448 1449 Donau 1450 Developer-internal name for the tax authority component. 1451 1452 **Use instead**: Tax authority 1453 1454 Terms to Use 1455 ------------ 1456 1457 Auditor 1458 Regulatory entity that certifies exchanges and oversees their operation. 1459 1460 Exchange Operator 1461 The entity/service that gives out digital cash in exchange for some 1462 other means of payment. 1463 1464 In some contexts, using "Issuer" could also be appropriate. 1465 When showing a balance breakdown, 1466 we can say "100 Eur (issued by exchange.euro.taler.net)". 1467 Sometimes we may also use the more generic term "Payment Service Provider" 1468 when the concept of an "Exchange" is still unclear to the reader. 1469 1470 Refund 1471 A refund is given by a merchant to the customer (rather the customer's wallet) 1472 and "undoes" a previous payment operation. 1473 1474 Payment 1475 The act of sending digital cash to a merchant to pay for an order. 1476 1477 Purchase 1478 Used to refer to the "result" of a payment, as in "view purchase". 1479 Use sparingly, as the word doesn't fit for all payments, such as donations. 1480 1481 Contract Terms 1482 Partially machine-readable representation of the merchant's obligation after the 1483 customer makes a payment. 1484 1485 Merchant 1486 Party that receives a payment. 1487 1488 Wallet 1489 Also "Taler Wallet". Software component that manages the user's digital cash 1490 and payments. 1491 1492 1493 Developer Glossary 1494 ================== 1495 1496 This glossary is meant for developers. It contains some terms that we usually do not 1497 use when talking to end users or even system administrators. 1498 1499 .. glossary:: 1500 :sorted: 1501 1502 absolute time 1503 method of keeping time in :term:`GNUnet` where the time is represented 1504 as the number of microseconds since 1.1.1970 (UNIX epoch). Called 1505 absolute time in contrast to :term:`relative time`. 1506 1507 aggregate 1508 the :term:`exchange` combines multiple payments received by the 1509 same :term:`merchant` into one larger :term:`wire transfer` to 1510 the respective merchant's :term:`bank` account 1511 1512 auditor 1513 trusted third party that verifies that the :term:`exchange` is operating correctly 1514 1515 bank 1516 traditional financial service provider who offers 1517 :term:`wire transfers <wire transfer>` between accounts 1518 1519 buyer 1520 individual in control of a Taler :term:`wallet`, usually using it to 1521 :term:`spend` the :term:`coins <coin>` on :term:`contracts <contract>` (see also :term:`customer`). 1522 1523 close 1524 operation an :term:`exchange` performs on a :term:`reserve` that has not been 1525 :term:`emptied <empty>` by :term:`withdraw` operations. When closing a reserve, the 1526 exchange wires the remaining funds back to the customer, minus a :term:`fee` 1527 for closing 1528 1529 customer 1530 individual that directs the buyer (perhaps the same individual) to make a purchase 1531 1532 coin 1533 coins are individual tokens representing a certain amount of value, also known as the :term:`denomination` of the coin 1534 1535 refresh commitment 1536 data that the wallet commits to during the :term:`melt` stage of the 1537 :term:`refresh` protocol where it 1538 has to prove to the :term:`exchange` that it is deriving the :term:`fresh` 1539 coins as specified by the Taler protocol. The commitment is verified 1540 probabilistically (see: :term:`kappa`) during the :term:`reveal` stage. 1541 1542 contract 1543 formal agreement between :term:`merchant` and :term:`customer` specifying the 1544 :term:`contract terms` and signed by the merchant and the :term:`coins <coin>` of the 1545 customer 1546 1547 contract terms 1548 the individual clauses specifying what the buyer is purchasing from the 1549 :term:`merchant` 1550 1551 denomination 1552 unit of currency, specifies both the currency and the face value of a :term:`coin`, 1553 as well as associated fees and validity periods 1554 1555 denomination key 1556 (RSA) key used by the exchange to certify that a given :term:`coin` is valid and of a 1557 particular :term:`denomination` 1558 1559 deposit 1560 operation by which a merchant passes coins to an exchange, expecting the 1561 exchange to credit his bank account in the future using an 1562 :term:`aggregate` :term:`wire transfer` 1563 1564 drain 1565 process by which an exchange operator takes the profits 1566 (from :term:`fees <fee>`) out of the escrow account and moves them into 1567 their regular business account 1568 1569 dirty 1570 a :term:`coin` is dirty if its public key may be known to an entity other than 1571 the customer, thereby creating the danger of some entity being able to 1572 link multiple transactions of coin's owner if the coin is not refreshed 1573 1574 empty 1575 a :term:`reserve` is being emptied when a :term:`wallet` is using the 1576 reserve's private key to :term:`withdraw` coins from it. This reduces 1577 the balance of the reserve. Once the balance reaches zero, we say that 1578 the reserve has been (fully) emptied. Reserves that are not emptied 1579 (which is the normal process) are :term:`closed <close>` by the exchange. 1580 1581 exchange 1582 Taler's payment service operator. Issues electronic coins during 1583 withdrawal and redeems them when they are deposited by merchants 1584 1585 expired 1586 Various operations come with time limits. In particular, denomination keys 1587 come with strict time limits for the various operations involving the 1588 coin issued under the denomination. The most important limit is the 1589 deposit expiration, which specifies until when wallets are allowed to 1590 use the coin in deposit or refreshing operations. There is also a "legal" 1591 expiration, which specifies how long the exchange keeps records beyond the 1592 deposit expiration time. This latter expiration matters for legal disputes 1593 in courts and also creates an upper limit for refreshing operations on 1594 special zombie coin 1595 1596 GNUnet 1597 Codebase of various libraries for a better Internet, some of which 1598 GNU Taler depends upon. 1599 1600 fakebank 1601 implementation of the :term:`bank` API in memory to be used only for test 1602 cases. 1603 1604 fee 1605 an :term:`exchange` charges various fees for its service. The different 1606 fees are specified in the protocol. There are fees per coin for 1607 :term:`withdrawing <withdraw>`, :term:`depositing <deposit>`, :term:`melting <melt>`, and 1608 :term:`refunding <refund>`. Furthermore, there are fees per wire transfer 1609 when a :term:`reserve` is :term:`closed <close>` 1610 and for :term:`aggregate` :term:`wire transfers <wire transfer>` 1611 to the :term:`merchant`. 1612 1613 fresh 1614 a :term:`coin` is fresh if its public key is only known to the customer 1615 1616 JSON 1617 JavaScript Object Notation (JSON) is a 1618 serialization format derived from the JavaScript language which is 1619 commonly used in the Taler protocol as the payload of HTTP requests 1620 and responses. 1621 1622 kappa 1623 security parameter used in the :term:`refresh` protocol. Defined to be 3. 1624 The probability of successfully evading the income transparency with the 1625 refresh protocol is 1:kappa. 1626 1627 libeufin 1628 Kotlin component that implements a regional currency bank and an 1629 adapter to communicate via EBICS with European core banking systems. 1630 1631 link 1632 specific step in the :term:`refresh` protocol that an exchange must offer 1633 to prevent abuse of the :term:`refresh` mechanism. The link step is 1634 not needed in normal operation, it just must be offered. 1635 1636 master key 1637 offline key used by the exchange to certify denomination keys and 1638 message signing keys 1639 1640 melt 1641 step of the :term:`refresh` protocol where a :term:`dirty` :term:`coin` 1642 is invalidated to be reborn :term:`fresh` in a subsequent 1643 :term:`reveal` step. 1644 1645 merchant 1646 party receiving payments (usually in return for goods or services) 1647 1648 message signing key 1649 key used by the exchange to sign online messages, other than coins 1650 1651 order 1652 offer made by the merchant to a wallet; pre-cursor to 1653 a contract where the wallet is not yet fixed. Turns 1654 into a :term:`contract` when a wallet claims the order. 1655 1656 owner 1657 a coin is owned by the entity that knows the private key of the coin 1658 1659 relative time 1660 method of keeping time in :term:`GNUnet` where the time is represented 1661 as a relative number of microseconds. Thus, a relative time specifies 1662 an offset or a duration, but not a date. Called relative time in 1663 contrast to :term:`absolute time`. 1664 1665 recoup 1666 Operation by which an exchange returns the value of coins affected 1667 by a :term:`revocation <revoke>` to their :term:`owner`, either by allowing the owner to 1668 withdraw new coins or wiring funds back to the bank account of the :term:`owner`. 1669 1670 planchet 1671 precursor data for a :term:`coin`. A planchet includes the coin's internal 1672 secrets (coin private key, blinding factor), but lacks the RSA signature 1673 of the :term:`exchange`. When :term:`withdrawing <withdraw>`, a :term:`wallet` 1674 creates and persists a planchet before asking the exchange to sign it to 1675 get the coin. 1676 1677 purchase 1678 Refers to the overall process of negotiating a :term:`contract` and then 1679 making a payment with :term:`coins <coin>` to a :term:`merchant`. 1680 1681 privacy policy 1682 Statement of an operator how they will protect the privacy of users. 1683 1684 proof 1685 Message that cryptographically demonstrates that a particular claim is correct. 1686 1687 proposal 1688 a list of :term:`contract terms` that has been completed and signed by the 1689 merchant backend. 1690 1691 refresh 1692 operation by which a :term:`dirty` :term:`coin` is converted into one or more 1693 :term:`fresh` coins. Involves :term:`melting <melt>` the :term:`dirty` coins and 1694 then :term:`revealing <reveal>` so-called :term:`transfer keys <transfer key>`. 1695 1696 refund 1697 operation by which a merchant steps back from the right to funds that he 1698 obtained from a :term:`deposit` operation, giving the right to the funds back 1699 to the customer 1700 1701 refund transaction id 1702 unique number by which a merchant identifies a :term:`refund`. Needed 1703 as refunds can be partial and thus there could be multiple refunds for 1704 the same :term:`purchase`. 1705 1706 reserve 1707 accounting mechanism used by the exchange to track customer funds 1708 from incoming :term:`wire transfers <wire transfer>`. A reserve is created whenever 1709 a customer wires money to the exchange using a well-formed public key 1710 in the subject. The exchange then allows the customer's :term:`wallet` 1711 to :term:`withdraw` up to the amount received in :term:`fresh` 1712 :term:`coins <coin>` from the reserve, thereby emptying the reserve. If a 1713 reserve is not emptied, the exchange will eventually :term:`close` it. 1714 1715 Other definition: Funds set aside for future use; either the balance of a customer at the 1716 exchange ready for withdrawal, or the funds kept in the exchange;s bank 1717 account to cover obligations from coins in circulation. 1718 1719 reveal 1720 step in the :term:`refresh` protocol where some of the transfer private 1721 keys are revealed to prove honest behavior on the part of the wallet. 1722 In the reveal step, the exchange returns the signed :term:`fresh` coins. 1723 1724 revoke 1725 exceptional operation by which an exchange withdraws a denomination from 1726 circulation, either because the signing key was compromised or because 1727 the exchange is going out of operation; unspent coins of a revoked 1728 denomination are subjected to recoup. 1729 1730 sharing 1731 users can share ownership of a :term:`coin` by sharing access to the coin's 1732 private key, thereby allowing all co-owners to spend the coin at any 1733 time. 1734 1735 spend 1736 operation by which a customer gives a merchant the right to deposit 1737 coins in return for merchandise 1738 1739 transfer key 1740 special cryptographic key used in the :term:`refresh` protocol, some of which 1741 are revealed during the :term:`reveal` step. Note that transfer keys have, 1742 despite the name, no relationship to :term:`wire transfers <wire transfer>`. They merely 1743 help to transfer the value from a :term:`dirty` coin to a :term:`fresh` coin 1744 1745 terms 1746 the general terms of service of an operator, possibly including 1747 the :term:`privacy policy`. Not to be confused with the 1748 :term:`contract terms` which are about the specific purchase. 1749 1750 transaction 1751 method by which ownership is exclusively transferred from one entity 1752 1753 user 1754 any individual using the Taler payment system 1755 (see :term:`customer`, :term:`buyer`, :term:`merchant`). 1756 1757 version 1758 Taler uses various forms of versioning. There is a database 1759 schema version (stored itself in the database, see \*-0000.sql) describing 1760 the state of the table structure in the database of an :term:`exchange`, 1761 :term:`auditor` or :term:`merchant`. There is a protocol 1762 version (CURRENT:REVISION:AGE, see GNU libtool) which specifies 1763 the network protocol spoken by an :term:`exchange` or :term:`merchant` 1764 including backwards-compatibility. And finally there is the software 1765 release version (MAJOR.MINOR.PATCH, see https://semver.org/) of 1766 the respective code base. 1767 1768 wallet 1769 software running on a customer's computer; withdraws, stores and 1770 spends coins 1771 1772 WebExtension 1773 Cross-browser API used to implement the GNU Taler wallet browser extension. 1774 1775 wire gateway 1776 API used by the exchange to talk with some real-time gross settlement system 1777 (core banking system, blockchain) to notice inbound credits wire transfers 1778 (during withdraw) and to trigger outbound debit wire transfers (primarily 1779 for deposits). 1780 1781 wire transfer 1782 a wire transfer is a method of sending funds between :term:`bank` accounts 1783 1784 wire transfer identifier 1785 Subject of a wire transfer from the exchange to a merchant; 1786 set by the aggregator to a random nonce which uniquely 1787 identifies the transfer. 1788 1789 withdraw 1790 operation by which a :term:`wallet` can convert funds from a :term:`reserve` to 1791 fresh coins 1792 1793 zombie 1794 :term:`coin` where the respective :term:`denomination key` is past its 1795 :term:`deposit` :term:`expiration <expired>` time, but which is still (again) valid 1796 for an operation because it was :term:`melted <melt>` while it was still 1797 valid, and then later again credited during a :term:`recoup` process 1798 1799 1800 1801 Developer Tools 1802 =============== 1803 1804 This section describes various internal programs to make life easier for the 1805 developer. 1806 1807 1808 taler-harness 1809 ------------- 1810 1811 **taler-harness deployment gen-coin-config** is a tool to simplify Taler configuration generation. 1812 1813 1814 **taler-harness deployment gen-coin-config** 1815 [**-min-amount**=**\ \ *VALUE*] 1816 [**-max-amount**=**\ \ *VALUE*]