ansible-taler-exchange

README (19479B)

---

 # Ansible Taler Playbooks
 
 ## Installing dependencies
 
 Depending on your local installation, you might need
 to install the following ansible collection:
 
 ```
 $ ansible-galaxy collection install community.postgresql
 ```
 
 Playbooks that touch vaulted variables need the vault password in
 "vault_pass.txt", which is gitignored and shipped only as
 "vault_pass.txt.gpg":
 
 ```
 $ gpg -d vault_pass.txt.gpg > vault_pass.txt
 ```
 
 ## Running the main Playbooks
 
 The canonical playbooks are run via shell scripts in the top-level
 directory.
 
 The operational helpers pass arguments after their positional arguments
 directly to `ansible-playbook`. For example:
 
 ```
 $ ./deploy spec --check --diff
 $ ./sanction-check spec sanctions.txt --diff
 $ BORG_PASSPHRASE=... ./restore spec archive-name --verbose
 ```
 
 Put the host/group first, followed by the sanction list or restore archive
 where required, then any Ansible options. `extract-borg-key` and
 `setup-pixel-borg` default to `spec` when called without arguments; supply
 an explicit target when passing options. Quote option values containing
 spaces as usual. Check-mode support depends on the playbook; restore
 explicitly rejects check mode. The `test` runner and `stage-activate` do
 not forward Ansible options.
 
 ### Local Podman deployment test
 
 Run `./test` to build a Debian Trixie container, provision the complete test
 deployment, repeat `setup.yml` with `--check --diff`, and exercise the main
 systemd units and HTTP APIs.  The script creates a disposable SSH key and
 removes the container when it exits.
 
 The container deliberately runs privileged because the hardened Taler systemd
 units need capabilities which Podman otherwise removes.  It also uses the
 pregenerated DH parameters and a container-only Certbot substitute which
 creates short-lived self-signed certificates: public ACME validation cannot
 reach the local test domains.  Neither shortcut is used by real deployments.
 
 ### Main setup (restore, deploy)
 
 Database restore is deliberately separate from normal deployment.  It is a
 fresh-host recovery operation, not an idempotent part of `deploy`, and it
 never chooses an archive implicitly.  First list the Borg repository and
 select the exact archive to restore:
 
 ```
 $ BORG_PASSPHRASE=... borg list ssh://borg@pixel.taler-systems.com/~/spec-backup
 $ BORG_PASSPHRASE=... ./restore spec 'archive-name-copied-from-list'
 ```
 
 The first argument must be one exact inventory hostname (not a group or host
 pattern).  The destination PostgreSQL cluster must contain only the standard
 `postgres` database and role.  The restore validates the compressed
 `root/postgres-backup.sql.gz` artifact, rejects dumps made by a newer
 PostgreSQL major version, restores all databases and roles, and verifies the
 exchange database before reporting success.  It does not deploy or start the
 application; inspect the recovered host and then run `deploy` separately.
 
 If SQL restore or verification fails, the partially restored cluster is
 considered tainted and must be reinitialized before another attempt.  The
 temporary database dump is removed, while a private error log is retained in
 `/var/tmp/taler-postgres-restore-*.log` on the target for diagnosis.
 
 The "deploy" script deploys the latest version of a system on a host.
 If you are root@rusty.taler-ops.ch, you may be able to:
 
 ```
 $ ./deploy rusty
 ```
 
 For TOPS production, replace the "rusty" with "spec" to use the actual secrets
 for the deployment. For this, you first need to decrypt them:
 
 ```
 $ ./contrib/decrypt inventories/host_vars/spec/prod-secrets.yml.gpg
 ```
 
 Make sure to NEVER commit the decrypted production secrets to Git.
 Instead, if you had to edit them, re-encrypt them to all admins:
 
 ```
 $ ./contrib/encrypt inventories/host_vars/spec/prod-secrets.yml
 ```
 
 ### Staging offline master key
 
 The `test-master.priv` offline master key is intentionally committed to this
 repository and referenced by `stage-offline.conf`.  It is a disposable test key
 for the staging exchange, where allowing repository users to sign staging
 configuration is intentional.  It must never be used for production, for an
 exchange holding funds of value, or as the basis for any security assumption.
 
 ### sanction-check
 
 This command imports and checks the latest sanction lists:
 
 ```
 $ ./sanction-check $TARGET $LIST
 ```
 
 where "$TARGET" is a host or group from "inventories/default" and $LIST
 is the path of the sanction list file on the local disk.
 
 Note that "SANCTION_LIST" means something else in the "exchange" role,
 where it is the name of a file under
 "roles/exchange/files/var/lib/taler-exchange/" that is deployed to the
 host and referenced from the exchange configuration.
 
 NOTE: this should still be further automated.
 
 
 ### External node monitoring
 
 To monitor an exchange node through `tsys-infra/monitoring`, first enroll it
 as an external monitoring client using that repository's
 `monitoring/README-external.md`. Retrieve both exported YAML files through
 an authenticated, encrypted channel. This repository configures only the
 exchange node; it does not connect to Sentol or enroll hosts itself.
 
 Place the exports in the exchange inventory host directory:
 
 ```
 inventories/host_vars/$HOST/monitoring-client.yml
 inventories/host_vars/$HOST/monitoring-client-secrets.yml
 ```
 
 The directory uses this repository's inventory hostname, such as `spec`.
 The identity inside the export remains the identity enrolled on Sentol;
 it need not equal that inventory alias. Use the complete exports without
 renaming their fields or combining them with other host variables.
 
 The proxy bind address defaults to `127.0.0.1` when omitted from the bundle.
 Set `monitoring_client.node_exporter.proxy_bind_address` to `"*"` (quoted
 in YAML) to listen on all IPv4 and IPv6 interfaces. This requires IPv6
 support on the node. Alternatively, use a literal IPv4 or IPv6 address
 assigned to the exchange node and reachable from the monitoring server.
 Hostnames and bare wildcard addresses (`0.0.0.0` and `::`) are rejected;
 use `"*"` for both address families. With `"*"`, the proxy and loopback
 backend must use different ports (normally 9100 and 9101). The proxy
 still requires mTLS, and the backend remains loopback-only. Keep the
 enrolled certificate identity unchanged.
 
 Commit both files, encrypting the entire secrets file with Ansible Vault
 before staging it:
 
 ```
 $ chmod 0600 inventories/host_vars/$HOST/monitoring-client-secrets.yml
 $ ansible-vault encrypt inventories/host_vars/$HOST/monitoring-client-secrets.yml
 ```
 
 Ansible uses the existing `vault_pass.txt` configuration. The committed
 secrets file must begin with `$ANSIBLE_VAULT;`; plaintext secrets are
 rejected when monitoring is enabled. Do not commit the plaintext export
 or the vault password. The Ansible controller needs Python 3 and OpenSSL
 for bundle validation. As with other vaulted host variables, Ansible may
 need the vault password even when monitoring management is disabled.
 
 Set this in the host's public inventory configuration, then deploy normally:
 
 ```yaml
 enable_monitoring: true
 ```
 
 ```
 $ ./deploy "$HOST"
 ```
 
 The `monitoring` role installs Debian's node_exporter with a loopback-only
 listener and systemd metrics, a dedicated `node-exporter-proxy` nginx
 service requiring Sentol's mTLS client identity, and buffered Fluent Bit
 JSON Lines forwarding over HTTPS/mTLS. All connection settings and certificate identities
 come from the bundle. TLS material lives in `/etc/taler-monitoring/tls`;
 private keys and the enclosing directory are accessible only to root.
 The root-running Fluent Bit service reads the journal and client key. The role
 requires Debian 13 amd64/arm64 and installs pinned Fluent Bit 5.1.2 from its signed
 upstream APT repository. The bundle declares `logs.protocol: jsonline` and an HTTPS
 `logs.url` ending in `/jsonline`; old RELP bundles fail preflight.
 
 Allow Sentol to reach the exported metrics proxy address and port
 (normally TCP 9100), and allow the node to reach the exported HTTPS log server
 (normally TCP 2514). Firewall and routing configuration is external to
 this role. The dedicated proxy leaves exchange webserver sites intact.
 
 Verify the local services with:
 
 ```
 $ systemctl is-active prometheus-node-exporter node-exporter-proxy fluent-bit
 $ curl --fail http://127.0.0.1:9101/metrics >/dev/null
 $ logger --tag external-monitoring-enrollment 'Exchange monitoring test'
 ```
 
 Use the exported backend address if it differs. Follow the external-client
 runbook to perform an authenticated scrape from Sentol, confirm anonymous
 scrapes fail, and find the test message in VictoriaLogs using the exported identity as
 `authenticated_host`. Fluent Bit metrics are available through the same mTLS proxy
 at `/fluent-bit/metrics`.
 
 Fluent Bit reads only new journal entries on first activation and resumes its saved
 cursor on later starts. Journald uses persistent storage without changing host-specific
 retention. `/var/lib/fluent-bit` retains the cursor and filesystem buffer, with a
 1 GiB output queue by default (`monitoring_fluent_bit_queue_limit`). Retryable failures
 are retried without a count limit; a full queue discards its oldest chunks. Permanent
 HTTP errors can discard records, and VictoriaLogs can accept a batch while rejecting
 individual malformed lines. Watch both collector drop counters and receiver errors.
 Lost responses can cause duplicates; this is not exactly-once or unconditional
 lossless delivery. Fluent Bit's own logs remain local to prevent feedback loops.
 
 Coordinate migration with Sentol's administrator: drain old RELP queues first, deploy
 the HTTPS receiver, then deploy this role with the new bundle. TCP 2514 is reused for
 HTTPS. First activation skips historical journal entries, so a cutover gap may be lost
 centrally. The role stops/removes rsyslog and rsyslog-relp without purging historical
 files, removes its forwarding configuration, and disables journal forwarding to syslog.
 Do not delete old spool files until any recovery is complete. Preserve the new cursor
 and buffer on rollback; restore old server and client configurations together.
 
 For renewal, retrieve both refreshed exports after `make deploy-sentol`,
 replace the local files, encrypt the complete new secrets file before
 staging it, commit both, and run `deploy` again. Do not reuse an old key
 with a renewed certificate. The role validates both pairs before deployment
 and reloads or restarts the affected services when their material changes.
 
 `enable_monitoring` defaults to false. False skips management and leaves
 installed monitoring running. To offboard, follow the Sentol runbook and,
 on the exchange node, stop and disable `node-exporter-proxy` and
 `prometheus-node-exporter` and `fluent-bit`, then remove the proxy unit
 and `/etc/taler-monitoring` (including its private keys) and run
 `systemctl daemon-reload`. Set the flag false and remove both inventory
 exports. This does not revoke previously issued certificates; complete
 the Sentol offboarding steps as well.
 
 ### Setting up backups (TOPS-only for now)
 
 First run:
 
 ```
 $ ./extract-borg-key $DEPLOYMENT
 ```
 
 The resulting SSH public key should be added to the borg-account
 of the host storing the backup. The borg_host inventory setting selects the target
 hostname.
 
 Once the SSH key is deployed and the backup has been initialized
 server-side (see admin-logs/pixel/03-borg.txt), start the daily
 backups via:
 
 ```
 $ ./start-borg-backups $DEPLOYMENT
 ```
 
 This will make a backup basically everything relevant to the
 deployment, **except** the exchange online signing keys. The
 backup will in particular include the system configuration
 and a full (gzip-compressed) snapshot of the database.  Thus,
 the backups should also suffice to diagnose problems.
 
 Backups are set to retain daily snapshots of the last 7 days,
 weekly snapshots for the last 4 weeks, and monthly snapshots
 for the last 6 months.
 
 
 ### Backup (right now)
 
 To run a backup "immediately" (instead of the daily regular
 backups), use:
 
 ```
 $ ./backup $DEPLOYMENT
 ```
 
 
 ### Rebooting (into a new kernel)
 
 This should be done via the 'reboot' playbook which can
 be invoked via the
 
 ```
 $ ./reboot $DEPLOYMENT
 ```
 
 script. The reboot playbook first stops all Taler services,
 then makes a backup, and then reboots. This should help us
 restore to another system in case the host does not come back
 online cleanly.
 
 
 ### Testing Locally
 
 With podman and ansible installed locally one can run:
 
 ```
 $ ./test
 ```
 
 This will begin building the Containerfile in this repo, which is a Debian
 base with systemd and a paswordless ssh server configured. Then container
 will start, binding port 8022 to 127.0.0.1 on the host. Finally
 the setup playbook will be run on the container via ssh.
 
 
 ## Playbooks
 
 ### backup
 
 Runs a backup "right now".
 
 ### borg-ssh-export
 
 Exports the SSH public keys needed at the remote host for backups.
 
 ### borg-start
 
 Enables the borg backup. Should be run after the SSH public keys
 exported via borg-ssh-export have been deployed on the receiving
 host.
 
 ### pixel-borg
 
 Enables receiving (!) backups from pixel. Adds the public key from
 pixel so we accept receiving borg backups from pixel.  Note that
 pixel still needs to be setup to send the backups.
 
 ### reboot
 
 Safely reboots the system by first stopping all Taler services,
 then making a backup and only then actually rebooting it.
 
 ### sanctionlist-check
 
 Imports the latest sanction list and checks all records against
 it.
 
 ### setup
 
 The main script that deploys our entire setup.
 
 ## Roles
 
 ### auditor
 
 Deploys the auditor.
 
 ### backup
 
 Runs the backup script, making a borg backup of the database and other key parts of the system.
 
 ### borg-ssh-export
 
 Exports the SSH public key that must be deployed on the host that is
 to receive the backup.
 
 ### borg-start
 
 Deploys the backup script and the cron job that runs it daily.
 
 ### cert
 
 Issues or renews a LetsEncrypt certificate with certbot and enables the
 nginx sites that use it. Included by the roles that own a vhost, not
 listed in a playbook directly.
 
 ### challenger
 
 Deploys the various challenger services for address verification.
 
 ### common_packages
 
 Installs the base system packages we need on all hosts.
 Sets up Taler package repo and installs Taler packages.
 
 ### database
 
 Installs PostgreSQL and the dependencies needed to administer it through
 Ansible, then ensures the database service is running.  Normal deployment
 never restores a backup.
 
 ### database_restore
 
 Implements the guarded fresh-cluster restore used only by `playbooks/restore.yml`
 and `restore`.
 
 ### devtesting
 
 Sets up a restricted account for faking wire transfers and inspecting
 challenger auth codes. Gated on "dangerously_enable_devtesting" and never
 to be enabled in production.
 
 ### exchange
 
 Deploys the Taler exchange, then hands over to "exchange_$DEPLOYMENT_KIND"
 for the site-specific configuration.
 
 ### exchange_tops
 
 Deployment-specific exchange configuration (denominations, KYC rules, fees,
 legal documents), selected by DEPLOYMENT_KIND.
 
 ### exchange-sanctionlist-import
 
 Imports a new sanction list and checks all existing records against it.
 
 ### libeufin-nexus
 
 Deploys libeufin-nexus which connects us to the bank.
 
 ### monitoring
 
 Provisions external node monitoring when `enable_monitoring` is true.
 
 ### pixel_borg
 
 Configures the host to hosting backups *from* pixel.
 
 ### reboot
 
 Reboots the system.
 
 ### stop_services
 
 Stops all Taler-related services. Useful for emergency stop and
 used as part of the reboot playbook.
 
 ### webserver
 
 Configures Nginx reverse proxy (main service, not individual subdomains).
 
 
 ## Checks
 
 ```
 $ ./contrib/test-fact-helpers.sh   # the /bin helpers that generate local facts
 ```
 
 Also run by the CI job in "contrib/ci/jobs/001-build".
 
 ## Deployment safety and recovery
 
 Normal deployment still upgrades system packages. It stops existing application
 services, targets, timers and sockets before package work, suppresses application
 starts through Debian's policy-rc.d mechanism, and starts applications
 only after all configuration and database migrations finish. Schedule deployment
 as downtime. During the system-wide upgrade, the temporary policy denies Taler
 service actions and delegates other actions to the existing administrator policy
 (or permits them if none existed). Application package tasks use policy_rc_d: 101.
 The original policy is restored even on failure. PostgreSQL and nginx are started
 as needed during provisioning.
 Cold exchange startup may spend several minutes generating denomination keys.
 post_deployment_http_retries defaults to 60 (with a five-second request timeout
 and one-second retry delay); systemd unit checks retain their shorter retry limit.
 
 A failed deployment leaves applications stopped without removing their boot
 enablement. Inspect the failed task, correct the problem, then rerun deploy.
 Do not reboot or manually start applications to bypass an unfinished migration.
 Optional services whose provisioning is skipped are restored if they were active
 when that deployment began. After a failed deployment, also inspect any optional
 services before returning the host to service.
 
 Setting dangerously_enable_devtesting to false revokes the managed login keys,
 sudo access and helper programs, terminates that account's sessions and disables
 the account. Its home-directory data is retained. Setting it true restores the
 managed account settings and access.
 
 The KYC attribute encryption key is preserved in a restricted secret include.
 Do not rotate this key as a substitute for moving it: changing it without a data
 migration would prevent decryption of existing attributes. Auditor authentication
 is stored in a root-only nginx include. Secret tasks suppress output and diffs,
 including when removing secrets from previously public configuration files.
 
 Backup and restore share borg_host and borg_repo inventory defaults, retaining
 ssh://borg@pixel.taler-systems.com/~/spec-backup for existing installations. Set
 borg_repo in host variables to use another already provisioned repository.
 Existing database_restore_borg_repository and database_restore_borg_host overrides
 remain supported. No archives are moved automatically.
 
 The backup script uses /root/.ssh/borg-config and leaves /root/.ssh/config intact.
 A lock protects the entire backup operation. Install the updated backup script
 between backup runs: an already running older script does not acquire this lock. If cron, backup or reboot
 encounters another backup, it exits nonzero (script status 75) without touching
 that run's snapshot. Retry after the existing backup finishes; a reboot playbook
 aborts before rebooting on this failure. Pruning and compaction are skipped after
 an unsuccessful or warning-producing archive creation.
 
 ## Regression checks
 
 Run python3 contrib/tests/test_backup.py for isolated concurrency, failure,
 cleanup and archive round-trip checks (requires Borg and Python Jinja2).
 The extended ./test also exercises real systemd shutdown, package-start
 suppression, a pre-2.20 Ansible dependency bootstrap, secret permissions,
 devtesting revocation, preservation of unrelated configuration, failed deployment
 and recovery. All fixtures run in the disposable container, never an inventory
 production host. Monitoring regressions also cover Vault-encrypted onboarding
 bundles, preflight failures before application shutdown, authenticated metrics,
 JSON Lines delivery, certificate renewal, idempotence, and check mode.