sandcastle-ng

README.md (9292B)

---

 # Introduction
 
 The sandcastle is a containerized deployment of GNU Taler
 
 It uses podman to build an image and run a single container that
 has systemd running inside.
 
 
 # Prerequisites
 
 You need (on your host system):
 
 * podman
 * bash
 
 The sandcastle-ng container exposes TCP ports that serve the APIs / Web
 interfaces of the respective GNU Taler service via HTTP.
 
 Some of these services expect to be accessed via a certain hostname,
 so it is recommended that you set up the following domain names:
 
 * `exchange.$MYDOMAIN` for taler-exchange
 * `backend.$MYDOMAIN` for taler-merchant
 * `bank.$MYDOMAIN` for libeufin-bank
 * `landing.$MYDOMAIN` for the demo landing page
 * `shop.$MYDOMAIN` for the demo blog shop page
 * `donations.$MYDOMAIN` for the demo donations page
 * `paivana.$MYDOMAIN` for the demo Paivana paywall
 
 Your host system should reverse-proxy HTTP(s) traffic to the respective service
 port for each of the services.
 
 
 # Upgrading Component Versions
 
 Run `./sandcastle-upgrade` to update the git tags in `buildconfig/$component.tag`
 to the latest upstream tag of the respective repository
 (taken from `buildconfig/$component.giturl`).
 
 By default only production tags (`vX.Y.Z`, `deb-vX.Y.Z-R`) are considered.
 Pass `--dev` to also consider dev tags (`vX.Y.Z-dev.N`).
 Pass `--dry` to only show what would change.
 Individual components can be given as arguments, e.g.
 
     ./sandcastle-upgrade --dev taler-exchange gnunet
 
 Components whose tag file pins something that is not a version tag
 (such as a branch name) are left alone, as are components without a
 `.giturl` file.
 
 To only *show* the latest upstream versions without changing anything,
 use `./print-latest-versions`.
 
 
 # Building the Container Image
 
 1. In `buildconfig/$component.tag` set the git tag you want to build.
 
 2. Run `./sandcastle-build` to build the Taler container.  The resulting container
    will be tagged as `taler-base-all`.
 
    By default `sandcastle-build` builds from the `Dockerfile`
    in the same directory as the script.
    Anyhow you can pass a different container file like this:
 
        ./sandcastle-build nightly.Dockerfile
 
 
 # Configuring the Deployment
 
 If you just want to play around with the Taler services inside the container,
 or want to get an integration development environment,
 there is a nice preset, see below.
 Thus in these cases no configuration is required
 and you can skip to "Running the Deployment" below.
 
 For Taler developers, it is recommended that for each sandcastle deployment,
 you clone the sandcastle-ng.git repository
 (https://git.taler.net/sandcastle-ng.git) and create a branch with
 deployment-specific changes.
 
 Currently there is not much configuration.
 
 The main adjustments to be made are:
 
 * scripts/demo/setup-sandcastle.sh has the currency on top of the file
 * sandcastle-run has variables for the ports that'll be exposed ("published") on
   the host.  They can be overwritten with environment variables
   (``TALER_SANDCASTLE_PORT_$COMPONENT``).
 
 To make adjustments for an environment without changing the
 ``setup-sandcastle.sh`` script, you can use an override file
 ``overrides/$YOUR_ENVIRONMENT``.  This is a bash script that is sourced at the
 beginning of ``setup-sandcastle.sh`` and can be used to customize
 environment-specific settings such as hostnames.
 
 In order to choose an override,
 set the ``SANDCASTLE_OVERRIDE_NAME`` environment variable
 before running `./sandcastle-run`:
 
     SANDCASTLE_OVERRIDE_NAME=$YOUR_ENVIRONMENT ./sandcastle-run
 
 
 The following variables are typically adjusted in an override file:
 
 * ``CURRENCY``: Currency of the deployment
 * ``MYDOMAIN``: Parent domain name for ``{bank,exchange,backend}`` services.
 * ``EXTERNAL_PORT``: The host port the services will be available on.
 * ``EXTERNAL_IP``: The host IP the services will be available on.
   For security reasons this defaults to localhost (127.0.0.1).
 * ``USE_INSECURE_SANDBOX_PASSWORDS``:
     If set to `1`, use password `sandbox` for all services instead of random ones.
     WARNING! Use only when listening to localhost!
 * ``PAIVANA_DESTINATION``: Upstream website that the Paivana paywall
   protects.  Defaults to `https://www.taler.net/`.  Note that this is an
   *external* server that the container proxies to, it is not served by the
   sandcastle itself.
 
 
 # Running the Deployment
 
 Run ``./sandcastle-run`` to run the single container.
 The container will be named `taler-sandcastle`.
 On the first run, wait until everything has been set up:
 
     ./sandcastle-logs
 
 
 Note that ``./sandcastle-run`` is just a wrapper around ``podman run``.
 If required, you can pass addtional arguments to ``./sandcastle-run``.
 These will be passed on to ``podman run``. Example:
 
     ./sandcastle-run --log-level=debug
 
 The running container publishes ports to the host as defined in ``./sandcastle-run``.
 You can manually verify these port mappings via ``podman port taler-sandcastle``.
 
 When starting sandcastle like
 
     SANDCASTLE_OVERRIDE_NAME=taler.localhost ./sandcastle-run
 
 the services will be available as
 
  - http://taler.localhost:4321
  - http://bank.taler.localhost:4321
  - http://shop.taler.localhost:4321
  - http://donations.taler.localhost:4321
  - http://backend.taler.localhost:4321  - the merchant backend
  - http://exchange.taler.localhost:4321
  - http://auditor.taler.localhost:4321
  - http://challenger.taler.localhost:4321
  - http://paivana.taler.localhost:4321/en/  - www.taler.net behind a Taler paywall
    (use the ``/en/`` entry point, as www.taler.net redirects ``/`` to its own domain)
 
 This gives a nice environment for integration testing.
 
 
 # Stopping the deployment
 
 For stopping the deployment simply run
 
 ```
 ./sandcastle-stop
 ```
 
 which is just a shortcut for ``podman stop taler-sandcastle``.
 
 
 # Watching Logs
 
 Run ``./sandcastle-logs`` to watch the logs within the container.
 You can pass addtional arguments to ``./sandcastle-logs``,
 which will be passed on to ``journalctl`` in the container.  Example:
 
     ./sandcastle-logs --output=cat
 
 
 # Poking Around
 
 You can poke around in a running sandcastle instance by running
 
 ```
 podman exec -it taler-sandcastle /bin/bash
 ```
 
 Or, as a shortcut:
 
 ```
 ./sandcastle-enter
 ```
 
 This will drop you into a shell inside the running container,
 where you have access to systemd, journalctl, etc.
 
 
 # Data Storage
 
 All persistent data is stored in a podman volume called
 _talerdata_.  You can see where it is in your filesystem
 by running ``podman volume inspect talerdata``.
 
 That volume also contains the postgres database files.
 
 ## Cleaning the Data Storage
 
 For cleaning the data storage run
 
     ./sandcastle-plow
 
 and for cleaning also the persistent storage run
 
     ./sandcastle-plow all
 
 
 # Provisioning Details
 
 The whole deployment is configured by a script
 mounted into the container as ``/provision/setup-sandcastle.sh``.
 This script will be run as a oneshot systemd service and will disable itself after
 the first success.
 
 To troubleshoot, run ``journalctl -u setup-sandcastle.service``.
 
 There are different setup scripts in the ``scripts/$SANDCASTLE_SETUP_NAME``
 folders. Specifically:
 
 * ``none`` does no setup at all
 * ``demo`` set up the usual Taler demo
 * TBD: ``regio`` is a currency conversion setup
 
 By default, ``demo`` is used.
 To mount a different provision script,
 set ``$SANDCASTLE_SETUP_NAME`` when running ``./sandcastle-run``,
 like this
 
     SANDCASTLE_SETUP_NAME=none ./sandcastle-run
 
 
 You can always manually run any of the provisioning script inside the container as
 ``/scripts/$SANDCASTLE_SETUP_NAME/setup-sandcastle.sh``.
 
 # Importing / exporting
 
 To migrate a sandcastle deployment, export / import can be used.
 
 ```
 # Requires taler-sandcastle to be running.
 # Export the sandcastle data to ./exported/
 ./sandcastle-export
 
 # Other host:
 # Request an import
 touch exported/import-request
 # Now the deployment script will pick up the data to import
 ./sandcastle-run
 ```
 
 # Neat Things That Already Work
 
 * Rebulding the base image is incremental, since we use layers.  If the tag
   of the exchange is changed, only the exchange and components that depend
   on it are rebuilt.
 * Inside the container, the service names resolve to localhost,
   and on localhost a reverse proxy with locally signed certificates
   ensures that services can talk to each other *within* the container
   by using their *public* base URL.
 
 
 # Future Extensions
 
 * Better way to access logs, better way to expose errors during provisioning
 * The Dockerfile should introduce nightly tags for debian packages it builds.
   Currently it just uses the latest defined version, which is confusing.
 * Deploy the Taler woocommerce plugin, wordpress plugin, Joomla plugin
 * Do self-tests of the deployment using the wallet CLI
 * Running the auditor
 * Running a currency conversion setup with multiple libeufin-bank instances
 * Instead of exposing HTTP ports, we could expose everything via unix domain sockets,
   avoiding port collision problems.
 * Instead of requiring the reverse proxy to handle TLS,
   the sandcastle container itself could do TLS termination with caddy.
 * To improve performance, allow connecting to an external database
 * Make it easy to import and export the persistent data
 * Extra tooling to checkpoint images/containers to revert to a previous
   state quickly.