commit ff04e066f6ac8a802a6b39a017d20fc10b7bf5c4
parent 515575073e194e3182776d1ca75a1d27f5c6c893
Author: Florian Dold <dold@taler.net>
Date: Mon, 7 Sep 2026 15:19:55 +0200
monitoring: add opt-in external node monitoring
Consume the tsys-infra monitoring onboarding exports from host variables,
requiring Ansible Vault encryption for the private keys. Validate the
bundle before stopping applications, then configure loopback node metrics,
a dedicated nginx mTLS proxy and durable RELP log forwarding.
Keep installed monitoring running when enable_monitoring is false, and
preserve its exporter and textfiles during legacy monitoring removal.
Diffstat:
15 files changed, 908 insertions(+), 10 deletions(-)
diff --git a/README b/README
@@ -125,6 +125,93 @@ host and referenced from the exchange configuration.
NOTE: this should still be further automated.
+### External node monitoring
+
+To monitor an exchange node through `tsys-infra/monitoring`, first enroll it
+as an external monitoring client using that repository's
+`monitoring/README-external.md`. Retrieve both exported YAML files through
+an authenticated, encrypted channel. This repository configures only the
+exchange node; it does not connect to Sentol or enroll hosts itself.
+
+Place the exports in the exchange inventory host directory:
+
+```
+inventories/host_vars/$HOST/monitoring-client.yml
+inventories/host_vars/$HOST/monitoring-client-secrets.yml
+```
+
+The directory uses this repository's inventory hostname, such as `spec`.
+The identity inside the export remains the identity enrolled on Sentol;
+it need not equal that inventory alias. Use the complete exports without
+renaming their fields or combining them with other host variables.
+
+Commit both files, encrypting the entire secrets file with Ansible Vault
+before staging it:
+
+```
+$ chmod 0600 inventories/host_vars/$HOST/monitoring-client-secrets.yml
+$ ansible-vault encrypt inventories/host_vars/$HOST/monitoring-client-secrets.yml
+```
+
+Ansible uses the existing `vault_pass.txt` configuration. The committed
+secrets file must begin with `$ANSIBLE_VAULT;`; plaintext secrets are
+rejected when monitoring is enabled. Do not commit the plaintext export
+or the vault password. The Ansible controller needs Python 3 and OpenSSL
+for bundle validation. As with other vaulted host variables, Ansible may
+need the vault password even when monitoring management is disabled.
+
+Set this in the host's public inventory configuration, then deploy normally:
+
+```yaml
+enable_monitoring: true
+```
+
+```
+$ ./deploy "$HOST"
+```
+
+The `monitoring` role installs Debian's node_exporter with a loopback-only
+listener and systemd metrics, a dedicated `node-exporter-proxy` nginx
+service requiring Sentol's mTLS client identity, and durable rsyslog
+RELP/TLS forwarding. All connection settings and certificate identities
+come from the bundle. TLS material lives in `/etc/taler-monitoring/tls`;
+private keys and the enclosing directory are accessible only to root.
+The role expects Debian's root-running rsyslog service to read those keys.
+
+Allow Sentol to reach the exported metrics proxy address and port
+(normally TCP 9100), and allow the node to reach the exported RELP server
+(normally TCP 2514). Firewall and routing configuration is external to
+this role. The dedicated proxy leaves exchange webserver sites intact.
+
+Verify the local services with:
+
+```
+$ systemctl is-active prometheus-node-exporter node-exporter-proxy rsyslog
+$ curl --fail http://127.0.0.1:9101/metrics >/dev/null
+$ logger --tag external-monitoring-enrollment 'Exchange monitoring test'
+```
+
+Use the exported backend address if it differs. Follow the external-client
+runbook to perform an authenticated scrape from Sentol, confirm anonymous
+scrapes fail, and find the test message in the remote log archive and
+VictoriaLogs.
+
+For renewal, retrieve both refreshed exports after `make deploy-sentol`,
+replace the local files, encrypt the complete new secrets file before
+staging it, commit both, and run `deploy` again. Do not reuse an old key
+with a renewed certificate. The role validates both pairs before deployment
+and reloads or restarts the affected services when their material changes.
+
+`enable_monitoring` defaults to false. False skips management and leaves
+installed monitoring running. To offboard, follow the Sentol runbook and,
+on the exchange node, stop and disable `node-exporter-proxy` and
+`prometheus-node-exporter`, remove `/etc/rsyslog.d/60-sentol-forward.conf`,
+validate with `rsyslogd -N1` and restart rsyslog, then remove the proxy unit
+and `/etc/taler-monitoring` (including its private keys) and run
+`systemctl daemon-reload`. Set the flag false and remove both inventory
+exports. This does not revoke previously issued certificates; complete
+the Sentol offboarding steps as well.
+
### remove-monitoring
The legacy monitoring stack is no longer deployed by this repository. To
@@ -136,6 +223,10 @@ $ ./remove-monitoring $DEPLOYMENT
```
This is destructive. It does not back up the local Prometheus or Alloy data.
+When the external monitoring proxy unit is installed, cleanup preserves
+its exporter service, package, configuration, and exporter textfile directory,
+independently of `enable_monitoring`. Other legacy Prometheus data is removed.
+It does not offboard the external monitoring client.
### Setting up backups (TOPS-only for now)
@@ -319,7 +410,8 @@ Deploys libeufin-nexus which connects us to the bank.
### monitoring
-Contains the cleanup tasks used by the remove-monitoring playbook.
+Provisions external node monitoring when `enable_monitoring` is true.
+Also contains the legacy cleanup tasks used by the remove-monitoring playbook.
### pixel_borg
@@ -402,4 +494,6 @@ The extended ./test also exercises real systemd shutdown, package-start
suppression, a pre-2.20 Ansible dependency bootstrap, secret permissions,
devtesting revocation, preservation of unrelated configuration, failed deployment
and recovery. All fixtures run in the disposable container, never an inventory
-production host.
+production host. Monitoring regressions also cover Vault-encrypted onboarding
+bundles, preflight failures before application shutdown, authenticated metrics,
+RELP delivery, certificate renewal, idempotence, check mode, and legacy cleanup.
diff --git a/contrib/tests/test_monitoring.py b/contrib/tests/test_monitoring.py
@@ -0,0 +1,288 @@
+#!/usr/bin/env python3
+"""Exercise external monitoring in the disposable deployment test container."""
+import argparse
+import copy
+import json
+import os
+from pathlib import Path
+import re
+import shutil
+import subprocess
+import tempfile
+import time
+
+REPO = Path(__file__).resolve().parents[2]
+IDENTITY = 'enrolled-exchange.example'
+SERVICES = ['prometheus-node-exporter', 'node-exporter-proxy', 'rsyslog']
+
+
+def run(*args, **kwargs):
+ return subprocess.run(list(map(str, args)), capture_output=True, text=True,
+ check=kwargs.pop('check', True), **kwargs)
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('container')
+ parser.add_argument('private_key')
+ args = parser.parse_args()
+
+ def container(*cmd, **kwargs):
+ return run('podman', 'exec', '-i', args.container, *cmd, **kwargs)
+
+ with tempfile.TemporaryDirectory(prefix='taler-monitoring-test-') as directory:
+ work = Path(directory)
+ inventory = work / 'inventory'
+ host_vars = inventory / 'host_vars/podman-localhost'
+ shutil.copytree(REPO / 'inventories/host_vars/podman-localhost', host_vars)
+ shutil.copytree(REPO / 'inventories/group_vars', inventory / 'group_vars')
+ shutil.copyfile(REPO / 'inventories/default', inventory / 'hosts')
+ vault_password = work / 'vault-password'
+ vault_password.write_text('disposable-monitoring-test-password\n')
+ vault_password.chmod(0o600)
+ pki = work / 'pki'
+ pki.mkdir(mode=0o700)
+
+ def openssl(*cmd):
+ return run('openssl', *cmd)
+
+ openssl('req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
+ '-nodes', '-keyout', pki / 'ca.key', '-out', pki / 'ca.cert', '-days', '2',
+ '-subj', '/CN=Disposable monitoring CA', '-addext', 'basicConstraints=critical,CA:TRUE')
+
+ def leaf(name, common_name, purpose):
+ openssl('req', '-new', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
+ '-nodes', '-keyout', pki / f'{name}.key', '-out', pki / f'{name}.csr',
+ '-subj', f'/CN={common_name}')
+ ext = pki / f'{name}.ext'
+ ext.write_text(f'basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\n'
+ f'extendedKeyUsage={purpose}\nsubjectAltName=DNS:{common_name}\n')
+ openssl('x509', '-req', '-in', pki / f'{name}.csr', '-CA', pki / 'ca.cert',
+ '-CAkey', pki / 'ca.key', '-CAcreateserial', '-days', '1',
+ '-extfile', ext, '-out', pki / f'{name}.cert')
+
+ leaf('client', IDENTITY, 'clientAuth')
+ leaf('server', IDENTITY, 'serverAuth')
+ leaf('sentol', 'sentol', 'clientAuth')
+ leaf('receiver', 'receiver.example', 'serverAuth')
+ secret_values = []
+
+ def bundle():
+ public = {'monitoring_client': {
+ 'identity': IDENTITY,
+ 'monitoring_ca_certificate': (pki / 'ca.cert').read_text(),
+ 'client_certificate': (pki / 'client.cert').read_text(),
+ 'server_certificate': (pki / 'server.cert').read_text(),
+ 'relp': {'server_address': '127.0.0.1', 'server_port': 12514,
+ 'server_name': 'receiver.example'},
+ 'node_exporter': {'backend_listen_address': '127.0.0.1:9101',
+ 'proxy_bind_address': '127.0.0.1', 'proxy_port': 9100,
+ 'prometheus_client_identity': 'sentol'}}}
+ secrets = {'monitoring_client_secrets': {
+ 'client_private_key': (pki / 'client.key').read_text(),
+ 'server_private_key': (pki / 'server.key').read_text()}}
+ secret_values.extend(secrets['monitoring_client_secrets'].values())
+ return public, secrets
+
+ def install_bundle(public, secrets, encrypt=True):
+ (host_vars / 'monitoring-client.yml').write_text(json.dumps(public))
+ path = host_vars / 'monitoring-client-secrets.yml'
+ path.write_text(json.dumps(secrets))
+ path.chmod(0o600)
+ if encrypt:
+ run('ansible-vault', 'encrypt', '--vault-password-file', vault_password, path,
+ env=dict(os.environ, ANSIBLE_CONFIG=str(REPO / 'test-ansible.cfg')))
+
+ env = dict(os.environ, ANSIBLE_CONFIG=str(REPO / 'test-ansible.cfg'), ANSIBLE_NOCOWS='1')
+ ansible = ['ansible-playbook', '-i', str(inventory / 'hosts'), '-l', 'podman-localhost',
+ '--user', 'root', '--private-key', args.private_key, '--vault-password-file',
+ str(vault_password), '-vvv', '--diff']
+
+ def invoke(playbook, enabled=True, check=False, failure=False):
+ command = ansible + ['-e', json.dumps({'enable_monitoring': enabled})]
+ if check:
+ command += ['--check']
+ result = run(*command, playbook, env=env, check=False)
+ output = result.stdout + result.stderr
+ assert 'BEGIN PRIVATE KEY' not in output, 'Private key appeared in Ansible output'
+ for secret in secret_values:
+ assert secret.splitlines()[1] not in output, 'Private key appeared in Ansible output'
+ if bool(result.returncode) != failure:
+ print(output)
+ raise AssertionError(f'Unexpected monitoring playbook status: {result.returncode}')
+ return output
+
+ role_play = work / 'monitoring.json'
+ role_play.write_text(json.dumps([{'hosts': 'all', 'gather_facts': False,
+ 'pre_tasks': [{'ansible.builtin.setup': {}, 'no_log': True}],
+ 'tasks': [{'ansible.builtin.include_role': {'name': 'monitoring'},
+ 'when': 'enable_monitoring | bool'}]}]))
+
+ def application_pid():
+ return container('systemctl', 'show', '-p', 'MainPID', 'taler-exchange-httpd').stdout
+
+ # Install and activate our own unrelated site; another regression's
+ # configuration file need not have been loaded by the running master.
+ container('tee', '/etc/nginx/conf.d/monitoring-regression-unrelated.conf',
+ input='server { listen 127.0.0.1:8098; return 200 "unrelated"; }\n')
+ container('nginx', '-t')
+ container('systemctl', 'reload', 'nginx')
+ container('curl', '--fail', '--silent', '--show-error', 'http://127.0.0.1:8098/')
+ pid = application_pid()
+ invoke(role_play, enabled=False)
+ assert container('test', '-e', '/etc/taler-monitoring', check=False).returncode != 0
+ invoke(REPO / 'playbooks/setup.yml', failure=True)
+ assert application_pid() == pid, 'Missing bundle stopped the exchange'
+ public, secrets = bundle()
+ install_bundle(public, secrets, encrypt=False)
+ invoke(REPO / 'playbooks/setup.yml', failure=True)
+ assert application_pid() == pid, 'Plaintext secrets stopped the exchange'
+ install_bundle(public, secrets)
+ (host_vars / 'monitoring-client.yml').unlink()
+ invoke(REPO / 'playbooks/setup.yml', failure=True)
+ install_bundle(public, secrets)
+ vault_password.write_text('incorrect-password\n')
+ invoke(REPO / 'playbooks/setup.yml', failure=True)
+ vault_password.write_text('disposable-monitoring-test-password\n')
+
+ for change in ['backend', 'identity', 'purpose', 'key', 'missing', 'injection']:
+ bad_public, bad_secrets = copy.deepcopy(public), copy.deepcopy(secrets)
+ if change == 'backend':
+ bad_public['monitoring_client']['node_exporter']['backend_listen_address'] = '0.0.0.0:9101'
+ elif change == 'identity':
+ bad_public['monitoring_client']['identity'] = 'wrong-host'
+ elif change == 'purpose':
+ bad_public['monitoring_client']['client_certificate'] = public['monitoring_client']['server_certificate']
+ elif change == 'key':
+ bad_secrets['monitoring_client_secrets']['client_private_key'] = secrets['monitoring_client_secrets']['server_private_key']
+ elif change == 'missing':
+ del bad_public['monitoring_client']['relp']
+ else:
+ bad_public['monitoring_client']['node_exporter']['prometheus_client_identity'] = 'sentol"; }'
+ install_bundle(bad_public, bad_secrets)
+ invoke(REPO / 'playbooks/setup.yml', failure=True)
+ assert application_pid() == pid, f'Invalid {change} stopped the exchange'
+ print('PASS: missing, plaintext, undecryptable and invalid bundles fail before shutdown', flush=True)
+
+ install_bundle(public, secrets)
+ invoke(role_play, check=True)
+ assert container('test', '-e', '/etc/taler-monitoring', check=False).returncode != 0
+ # A legacy exporter may already own port 9100. Reconfiguration must
+ # move it to loopback before the dedicated TLS proxy takes that port.
+ container('apt-get', 'install', '-y', '--no-install-recommends', 'prometheus-node-exporter')
+ container('systemctl', 'start', 'prometheus-node-exporter')
+ invoke(role_play)
+
+ remote = '/tmp/taler-monitoring-test'
+ container('install', '-d', '-m', '0700', remote)
+ run('podman', 'cp', pki, f'{args.container}:{remote}/pki')
+
+ def proxy(kind='sentol', path='/metrics', status=False):
+ command = ['curl', '--silent', '--show-error', '--max-time', '10', '--noproxy', '*',
+ '--cacert', f'{remote}/pki/ca.cert', '--resolve', f'{IDENTITY}:9100:127.0.0.1']
+ if kind:
+ command += ['--cert', f'{remote}/pki/{kind}.cert', '--key', f'{remote}/pki/{kind}.key']
+ if status:
+ command += ['--output', '/dev/null', '--write-out', '%{http_code}']
+ else:
+ command += ['--fail']
+ return container(*command, f'https://{IDENTITY}:9100{path}', check=False)
+
+ def verify_metrics():
+ for _ in range(30):
+ result = proxy()
+ if result.returncode == 0:
+ break
+ time.sleep(1)
+ assert result.returncode == 0, result.stderr
+ assert 'node_systemd_unit_state{' in result.stdout
+ assert 'taler-exchange-httpd.service' in result.stdout
+ assert proxy('client', status=True).stdout == '403'
+ assert proxy(None, status=True).stdout in ['400', '403']
+ assert proxy(path='/other', status=True).stdout == '404'
+ listeners = container('ss', '-lnt').stdout
+ assert '127.0.0.1:9101' in listeners
+ assert '0.0.0.0:9101' not in listeners and '[::]:9101' not in listeners
+ assert container('curl', '--fail', '--max-time', '5', 'http://127.0.0.1:9100/metrics',
+ check=False).returncode != 0
+ container('systemctl', 'is-active', '--quiet', *SERVICES)
+
+ verify_metrics()
+ for path, expected in [('tls', '700'), ('tls/client.key.pem', '600'), ('tls/server.key.pem', '600')]:
+ assert container('stat', '-c', '%a:%U:%G', f'/etc/taler-monitoring/{path}').stdout.strip() == f'{expected}:root:root'
+ assert container('su', '-s', '/bin/sh', 'nobody', '-c',
+ 'test -r /etc/taler-monitoring/tls/client.key.pem', check=False).returncode != 0
+ receiver = f'''module(load="imrelp" tls.tlsLib="openssl")
+ruleset(name="received") {{ action(type="omfile" file="{remote}/received.log") }}
+input(type="imrelp" address="127.0.0.1" port="12514" ruleset="received"
+ tls="on" tls.caCert="{remote}/pki/ca.cert"
+ tls.myCert="{remote}/pki/receiver.cert" tls.myPrivKey="{remote}/pki/receiver.key"
+ tls.authMode="name" tls.permittedPeer=["{IDENTITY}"])
+'''
+ container('tee', f'{remote}/receiver.conf', input=receiver)
+ container('rsyslogd', '-N1', '-f', f'{remote}/receiver.conf')
+
+ def receiver_start():
+ container('systemd-run', '--unit=monitoring-test-receiver', '--collect',
+ 'rsyslogd', '-n', '-f', f'{remote}/receiver.conf', '-i', f'{remote}/receiver.pid')
+
+ def log_delivery(message, emit=True):
+ if emit:
+ container('logger', '--tag', 'monitoring-regression', message)
+ for _ in range(45):
+ received = container('cat', f'{remote}/received.log', check=False).stdout
+ if message in received:
+ return
+ time.sleep(1)
+ raise AssertionError('RELP test message was not delivered')
+
+ receiver_start()
+ try:
+ log_delivery('initial-monitoring-enrollment')
+ print('PASS: mTLS metrics, client allowlist, systemd metrics, permissions and RELP delivery', flush=True)
+ container('systemctl', 'stop', 'monitoring-test-receiver')
+ container('logger', '--tag', 'monitoring-regression', 'queued-during-receiver-outage')
+ time.sleep(2)
+ container('systemctl', 'restart', 'rsyslog')
+ receiver_start()
+ log_delivery('queued-during-receiver-outage', emit=False)
+ print('PASS: RELP queue survives receiver outage and rsyslog restart', flush=True)
+ before = container('systemctl', 'show', '-p', 'MainPID', *SERVICES).stdout
+ output = invoke(role_play)
+ assert re.search(r'changed=0\s', output), 'Second monitoring deployment was not idempotent'
+ invoke(role_play, check=True)
+ invoke(role_play, enabled=False)
+ assert container('systemctl', 'show', '-p', 'MainPID', *SERVICES).stdout == before
+
+ leaf('client', IDENTITY, 'clientAuth')
+ leaf('server', IDENTITY, 'serverAuth')
+ public, secrets = bundle()
+ install_bundle(public, secrets)
+ invoke(role_play)
+ verify_metrics()
+ log_delivery('renewed-monitoring-certificates')
+ assert container('cat', '/etc/taler-monitoring/tls/server.cert.pem').stdout == (pki / 'server.cert').read_text()
+ print('PASS: idempotence, check mode, disabled management and certificate renewal', flush=True)
+
+ # Exercise the real setup entrypoint with monitoring enabled.
+ invoke(REPO / 'playbooks/setup.yml')
+ verify_metrics()
+ container('mkdir', '-p', '/var/lib/prometheus/metrics2')
+ container('touch', '/var/lib/prometheus/metrics2/legacy-data',
+ '/var/lib/prometheus/node-exporter/keep.prom')
+ invoke(REPO / 'playbooks/remove-monitoring.yml', enabled=False)
+ assert container('test', '-e', '/var/lib/prometheus/metrics2', check=False).returncode != 0
+ container('test', '-f', '/var/lib/prometheus/node-exporter/keep.prom')
+ verify_metrics()
+ log_delivery('monitoring-survives-legacy-removal')
+ container('curl', '--fail', '--silent', '--show-error', '--unix-socket',
+ '/var/run/taler-exchange/httpd/exchange-http.sock', 'http://localhost/config')
+ container('curl', '--fail', '--silent', '--show-error', 'http://127.0.0.1:8098/')
+ print('PASS: normal deployment and legacy cleanup preserve monitoring and exchange endpoints', flush=True)
+ finally:
+ container('systemctl', 'stop', 'monitoring-test-receiver', check=False)
+ container('rm', '-rf', remote)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/inventories/group_vars/all/defaults.yml b/inventories/group_vars/all/defaults.yml
@@ -6,6 +6,10 @@ deploy_auditor: true
# Deploy challenger?
deploy_challenger: false
+# Provision the external monitoring bundle enrolled in tsys-infra/monitoring.
+# False skips management; it does not stop previously installed monitoring.
+enable_monitoring: false
+
# If true, use EBICS keys from that were externally created.
ebics_keys_external: false
diff --git a/playbooks/setup.yml b/playbooks/setup.yml
@@ -8,6 +8,12 @@
ansible.builtin.setup:
no_log: true
+ - name: Validate the external monitoring bundle before stopping applications
+ ansible.builtin.include_role:
+ name: monitoring
+ tasks_from: preflight
+ when: enable_monitoring | default(false) | bool
+
- name: Reject the removed in-deployment restore switch
ansible.builtin.assert:
that: not (enable_restore_backup | default(false) | bool)
@@ -89,6 +95,11 @@
ansible.builtin.include_role:
name: webserver
+ - name: Configure monitoring
+ ansible.builtin.include_role:
+ name: monitoring
+ when: enable_monitoring | default(false) | bool
+
- name: Configure database
ansible.builtin.include_role:
name: database
diff --git a/roles/monitoring/files/node-exporter-proxy.service b/roles/monitoring/files/node-exporter-proxy.service
@@ -0,0 +1,30 @@
+[Unit]
+Description=nginx mTLS proxy for Prometheus node_exporter
+Requires=prometheus-node-exporter.service
+After=network-online.target prometheus-node-exporter.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+ExecStartPre=/usr/sbin/nginx -t -c /etc/taler-monitoring/nginx.conf
+ExecStart=/usr/sbin/nginx -c /etc/taler-monitoring/nginx.conf -g 'daemon off;'
+ExecReload=/bin/kill -HUP $MAINPID
+KillSignal=SIGQUIT
+TimeoutStopSec=5s
+Restart=on-failure
+RestartSec=5s
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=full
+ProtectControlGroups=true
+ProtectKernelLogs=true
+ProtectKernelModules=true
+ProtectKernelTunables=true
+RestrictSUIDSGID=true
+LockPersonality=true
+MemoryDenyWriteExecute=true
+SystemCallArchitectures=native
+
+[Install]
+WantedBy=multi-user.target
diff --git a/roles/monitoring/files/validate-bundle.py b/roles/monitoring/files/validate-bundle.py
@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+"""Validate external monitoring exports without disclosing their contents."""
+import ipaddress
+import json
+import re
+from pathlib import Path
+import subprocess
+import sys
+import tempfile
+
+
+def require(condition, message):
+ if not condition:
+ raise ValueError(message)
+
+
+def identity(value):
+ return isinstance(value, str) and re.fullmatch(r'[A-Za-z0-9_.-]+', value)
+
+
+def port(value):
+ return type(value) is int and 0 < value < 65536
+
+
+def address(value):
+ require(isinstance(value, str) and '%' not in value,
+ 'Monitoring proxy bind address must be an IP address without a scope ID.')
+ try:
+ # The proxy template supplies brackets for IPv6.
+ return ipaddress.ip_address(value.removeprefix('[').removesuffix(']'))
+ except ValueError:
+ raise ValueError('Monitoring proxy bind address must be an IP address.') from None
+
+
+def openssl(*args):
+ result = subprocess.run(['openssl', *map(str, args)], capture_output=True)
+ require(result.returncode == 0, 'Monitoring certificate or private key validation failed.')
+ return result.stdout
+
+
+def validate(bundle):
+ public = bundle['public']['monitoring_client']
+ secret = bundle['secrets']['monitoring_client_secrets']
+ node = public['node_exporter']
+ relp = public['relp']
+ require(identity(public['identity']) and identity(node['prometheus_client_identity']),
+ 'Invalid monitoring certificate identity.')
+ require(identity(relp['server_name']), 'Invalid RELP server identity.')
+ target = relp['server_address']
+ require(isinstance(target, str) and re.fullmatch(r'[A-Za-z0-9_.:-]+', target),
+ 'Invalid RELP server address.')
+ require(port(relp['server_port']) and port(node['proxy_port']), 'Invalid monitoring port.')
+ backend = node['backend_listen_address']
+ require(isinstance(backend, str), 'Invalid node_exporter backend address.')
+ match = re.fullmatch(r'(127\.0\.0\.1|\[::1\]):([0-9]+)', backend)
+ require(match and 0 < int(match[2]) < 65536,
+ 'The node_exporter backend must use 127.0.0.1 or [::1] and a valid port.')
+ bind = address(node['proxy_bind_address'])
+ require(not bind.is_unspecified and not bind.is_multicast,
+ 'The monitoring proxy must bind a specific unicast address.')
+ require(not (bind.is_loopback and node['proxy_port'] == int(match[2])),
+ 'Monitoring proxy and backend listeners must not conflict.')
+
+ # TemporaryDirectory is private (0700); private files are created as 0600.
+ # Validate the complete pair before Ansible replaces any live material.
+ with tempfile.TemporaryDirectory(prefix='taler-monitoring-') as directory:
+ root = Path(directory)
+ values = {'ca': public['monitoring_ca_certificate'],
+ 'client': public['client_certificate'], 'server': public['server_certificate'],
+ 'client-key': secret['client_private_key'], 'server-key': secret['server_private_key']}
+ for name, value in values.items():
+ require(isinstance(value, str) and value.strip(), 'Missing monitoring TLS material.')
+ path = root / name
+ path.touch(mode=0o600)
+ path.write_text(value.strip() + '\n')
+ for kind, purpose, eku in [('client', 'sslclient', 'TLS Web Client Authentication'),
+ ('server', 'sslserver', 'TLS Web Server Authentication')]:
+ cert, key = root / kind, root / (kind + '-key')
+ openssl('verify', '-CAfile', root / 'ca', '-purpose', purpose,
+ '-verify_hostname', public['identity'], cert)
+ subject = openssl('x509', '-in', cert, '-noout', '-subject', '-nameopt', 'RFC2253')
+ require(subject.decode().strip() == 'subject=CN=' + public['identity'],
+ 'Monitoring certificate subject does not match the enrolled identity.')
+ sans = openssl('x509', '-in', cert, '-noout', '-ext', 'subjectAltName')
+ require(('DNS:' + public['identity']) in sans.decode().splitlines()[1].strip().split(', '),
+ 'Monitoring certificate SAN does not contain the enrolled identity.')
+ extensions = openssl('x509', '-in', cert, '-noout', '-ext', 'extendedKeyUsage')
+ require(extensions.decode().splitlines()[1].strip() == eku,
+ 'Monitoring leaf must have exactly its designated TLS purpose.')
+ certificate_key = openssl('x509', '-in', cert, '-pubkey', '-noout')
+ private_key_public = openssl('pkey', '-in', key, '-passin', 'pass:', '-pubout')
+ require(certificate_key == private_key_public,
+ 'Monitoring certificate does not match its private key.')
+
+
+def main():
+ try:
+ if len(sys.argv) == 3 and sys.argv[1] == '--files':
+ directory = Path(sys.argv[2])
+ for name in ['monitoring-client.yml', 'monitoring-client-secrets.yml']:
+ require((directory / name).is_file(),
+ 'Both monitoring-client.yml and monitoring-client-secrets.yml are required in host_vars/<host>/.')
+ with (directory / 'monitoring-client-secrets.yml').open('rb') as source:
+ require(source.readline().startswith(b'$ANSIBLE_VAULT;'),
+ 'Encrypt the complete monitoring-client-secrets.yml with ansible-vault before deployment.')
+ else:
+ validate(json.load(sys.stdin))
+ except ValueError as error:
+ # JSON parse errors can contain user-supplied data; never print them.
+ print('Invalid monitoring bundle JSON.' if isinstance(error, json.JSONDecodeError) else str(error))
+ return 1
+ except (KeyError, TypeError, IndexError):
+ print('Monitoring bundle is missing required fields or contains invalid field types.')
+ return 1
+ except OSError:
+ print('Cannot read monitoring files or run openssl on the Ansible controller.')
+ return 1
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/roles/monitoring/handlers/main.yml b/roles/monitoring/handlers/main.yml
@@ -1,4 +1,46 @@
---
+- name: Restart monitoring node_exporter
+ ansible.builtin.systemd_service:
+ name: prometheus-node-exporter
+ state: restarted
+ when: not ansible_check_mode
+
+- name: Validate monitoring proxy before reload or restart
+ ansible.builtin.command: /usr/sbin/nginx -t -c /etc/taler-monitoring/nginx.conf
+ changed_when: false
+ listen:
+ - Reload monitoring proxy
+ - Restart monitoring proxy
+ when: not ansible_check_mode
+
+- name: Reload validated monitoring proxy
+ ansible.builtin.systemd_service:
+ name: node-exporter-proxy
+ state: reloaded
+ listen: Reload monitoring proxy
+ when: not ansible_check_mode
+
+- name: Restart validated monitoring proxy
+ ansible.builtin.systemd_service:
+ name: node-exporter-proxy
+ daemon_reload: true
+ state: restarted
+ listen: Restart monitoring proxy
+ when: not ansible_check_mode
+
+- name: Validate monitoring rsyslog before restart
+ ansible.builtin.command: /usr/sbin/rsyslogd -N1
+ changed_when: false
+ listen: Restart monitoring rsyslog
+ when: not ansible_check_mode
+
+- name: Restart validated monitoring rsyslog
+ ansible.builtin.systemd_service:
+ name: rsyslog
+ state: restarted
+ listen: Restart monitoring rsyslog
+ when: not ansible_check_mode
+
- name: Validate nginx before restart
ansible.builtin.command: nginx -c /etc/nginx/nginx.conf -t
changed_when: false
diff --git a/roles/monitoring/tasks/disable.yml b/roles/monitoring/tasks/disable.yml
@@ -1,4 +1,9 @@
---
+- name: Detect the external monitoring installation independently of inventory
+ ansible.builtin.stat:
+ path: /etc/systemd/system/node-exporter-proxy.service
+ register: monitoring_external_installation
+
- name: Get the list of services
service_facts:
@@ -14,4 +19,6 @@
- prometheus-postgres-exporter.service
- prometheus-alertmanager.service
- prometheus.service
- when: item in ansible_facts["services"]
+ when:
+ - item in ansible_facts["services"]
+ - item != 'prometheus-node-exporter.service' or not monitoring_external_installation.stat.exists
diff --git a/roles/monitoring/tasks/main.yml b/roles/monitoring/tasks/main.yml
@@ -0,0 +1,141 @@
+---
+- name: Validate the external monitoring bundle
+ ansible.builtin.include_tasks: preflight.yml
+
+- name: Discover existing monitoring services for check mode
+ ansible.builtin.service_facts:
+ when: ansible_check_mode
+
+- name: Install monitoring client packages without starting default listeners
+ ansible.builtin.apt:
+ name:
+ - nginx
+ - prometheus-node-exporter
+ - rsyslog
+ - rsyslog-relp
+ state: present
+ update_cache: true
+ cache_valid_time: 3600
+ install_recommends: false
+ policy_rc_d: 101
+
+- name: Create monitoring configuration and TLS directories
+ ansible.builtin.file:
+ path: "{{ item.path }}"
+ state: directory
+ owner: root
+ group: root
+ mode: "{{ item.mode }}"
+ loop:
+ - { path: /etc/taler-monitoring, mode: '0755' }
+ - { path: /etc/taler-monitoring/tls, mode: '0700' }
+
+- name: Install monitoring TLS material
+ ansible.builtin.copy:
+ content: "{{ item.content | trim }}\n"
+ dest: "/etc/taler-monitoring/tls/{{ item.name }}"
+ owner: root
+ group: root
+ mode: "{{ item.mode }}"
+ loop:
+ - name: ca.cert.pem
+ content: "{{ monitoring_public_bundle.monitoring_client.monitoring_ca_certificate }}"
+ mode: '0644'
+ - name: client.cert.pem
+ content: "{{ monitoring_public_bundle.monitoring_client.client_certificate }}"
+ mode: '0644'
+ - name: client.key.pem
+ content: "{{ monitoring_secret_bundle.monitoring_client_secrets.client_private_key }}"
+ mode: '0600'
+ - name: server.cert.pem
+ content: "{{ monitoring_public_bundle.monitoring_client.server_certificate }}"
+ mode: '0644'
+ - name: server.key.pem
+ content: "{{ monitoring_secret_bundle.monitoring_client_secrets.server_private_key }}"
+ mode: '0600'
+ no_log: true
+ diff: false
+ notify:
+ - Reload monitoring proxy
+ - Restart monitoring rsyslog
+
+- name: Create the node_exporter textfile directory
+ ansible.builtin.file:
+ path: /var/lib/prometheus/node-exporter
+ state: directory
+ owner: prometheus
+ group: prometheus
+ mode: '0755'
+ when: not ansible_check_mode or 'prometheus-node-exporter.service' in ansible_facts['services']
+
+- name: Configure the private node_exporter listener and systemd collector
+ ansible.builtin.template:
+ src: node-exporter-default.j2
+ dest: /etc/default/prometheus-node-exporter
+ owner: root
+ group: root
+ mode: '0644'
+ notify: Restart monitoring node_exporter
+
+- name: Configure the dedicated monitoring proxy
+ ansible.builtin.template:
+ src: nginx.conf.j2
+ dest: /etc/taler-monitoring/nginx.conf
+ owner: root
+ group: root
+ mode: '0644'
+ validate: /usr/sbin/nginx -t -c %s
+ notify: Reload monitoring proxy
+
+- name: Install the dedicated monitoring proxy unit
+ ansible.builtin.copy:
+ src: node-exporter-proxy.service
+ dest: /etc/systemd/system/node-exporter-proxy.service
+ owner: root
+ group: root
+ mode: '0644'
+ notify: Restart monitoring proxy
+
+- name: Create the persistent RELP queue directory
+ ansible.builtin.file:
+ path: /var/spool/rsyslog
+ state: directory
+ owner: root
+ group: root
+ mode: '0700'
+
+- name: Configure authenticated RELP forwarding
+ ansible.builtin.template:
+ src: rsyslog-forward.conf.j2
+ dest: /etc/rsyslog.d/60-sentol-forward.conf
+ owner: root
+ group: root
+ mode: '0644'
+ validate: /usr/sbin/rsyslogd -N1 -f %s
+ notify: Restart monitoring rsyslog
+
+- name: Validate the complete rsyslog configuration before activation
+ ansible.builtin.command: /usr/sbin/rsyslogd -N1
+ changed_when: false
+ when: not ansible_check_mode
+
+- name: Load the configured monitoring units
+ ansible.builtin.systemd_service:
+ daemon_reload: true
+ when: not ansible_check_mode
+
+# Move an existing legacy exporter off port 9100 before starting the proxy.
+# This also loads renewed certificate pairs before reporting success.
+- name: Apply monitoring configuration before enabling services
+ ansible.builtin.meta: flush_handlers
+
+- name: Enable monitoring services after configuration
+ ansible.builtin.systemd_service:
+ name: "{{ item }}"
+ state: started
+ enabled: true
+ loop:
+ - prometheus-node-exporter
+ - node-exporter-proxy
+ - rsyslog
+ when: not ansible_check_mode or (item ~ '.service') in ansible_facts['services']
diff --git a/roles/monitoring/tasks/preflight.yml b/roles/monitoring/tasks/preflight.yml
@@ -0,0 +1,68 @@
+---
+- name: Check monitoring bundle files and whole-file Vault encryption
+ ansible.builtin.command:
+ argv:
+ - "{{ ansible_playbook_python }}"
+ - "{{ role_path }}/files/validate-bundle.py"
+ - --files
+ - "{{ inventory_dir }}/host_vars/{{ inventory_hostname }}"
+ delegate_to: localhost
+ become: false
+ register: monitoring_file_check
+ changed_when: false
+ failed_when: false
+ check_mode: false
+
+- name: Require both monitoring exports with Vault-encrypted secrets
+ ansible.builtin.assert:
+ that: monitoring_file_check.rc == 0
+ fail_msg: "{{ monitoring_file_check.stdout }}"
+ quiet: true
+
+- name: Load the monitoring exports without logging credentials
+ block:
+ - name: Load public monitoring configuration
+ ansible.builtin.include_vars:
+ file: "{{ inventory_dir }}/host_vars/{{ inventory_hostname }}/monitoring-client.yml"
+ name: monitoring_public_bundle
+ no_log: true
+
+ - name: Decrypt monitoring credentials
+ ansible.builtin.include_vars:
+ file: "{{ inventory_dir }}/host_vars/{{ inventory_hostname }}/monitoring-client-secrets.yml"
+ name: monitoring_secret_bundle
+ no_log: true
+
+ - name: Validate monitoring settings and certificate pairs on the controller
+ ansible.builtin.command:
+ argv:
+ - "{{ ansible_playbook_python }}"
+ - "{{ role_path }}/files/validate-bundle.py"
+ stdin: >-
+ {{ {'public': monitoring_public_bundle, 'secrets': monitoring_secret_bundle} | to_json }}
+ delegate_to: localhost
+ become: false
+ register: monitoring_bundle_check
+ changed_when: false
+ failed_when: false
+ check_mode: false
+ no_log: true
+
+ # The validator prints only fixed diagnostics, never supplied values or PEMs.
+ - name: Require a valid external monitoring bundle
+ ansible.builtin.assert:
+ that: monitoring_bundle_check.rc == 0
+ fail_msg: "{{ monitoring_bundle_check.stdout }}"
+ quiet: true
+
+ rescue:
+ - name: Report an unusable monitoring bundle
+ ansible.builtin.fail:
+ msg: >-
+ Cannot load or validate the monitoring exports. Check the exported YAML,
+ Vault password, certificate identities, purposes, validity and key pairs.
+
+- name: Require a Debian-family monitoring client
+ ansible.builtin.assert:
+ that: ansible_facts['os_family'] == 'Debian'
+ quiet: true
diff --git a/roles/monitoring/tasks/remove.yml b/roles/monitoring/tasks/remove.yml
@@ -51,13 +51,10 @@
- name: Purge legacy monitoring packages
ansible.builtin.apt:
- name:
- - alloy
- - prometheus
- - prometheus-alertmanager
- - prometheus-nginx-exporter
- - prometheus-node-exporter
- - prometheus-postgres-exporter
+ name: >-
+ {{ ['alloy', 'prometheus', 'prometheus-alertmanager',
+ 'prometheus-nginx-exporter', 'prometheus-postgres-exporter']
+ + ([] if monitoring_external_installation.stat.exists else ['prometheus-node-exporter']) }}
state: absent
purge: true
autoremove: true
@@ -78,6 +75,26 @@
- /etc/prometheus
- /var/lib/alloy
- /var/lib/prometheus
+ when: >-
+ not monitoring_external_installation.stat.exists or
+ item not in ['/etc/default/prometheus-node-exporter', '/var/lib/prometheus']
+
+- name: Find legacy Prometheus data alongside the retained exporter directory
+ ansible.builtin.find:
+ paths: /var/lib/prometheus
+ file_type: any
+ hidden: true
+ excludes: node-exporter
+ register: monitoring_legacy_prometheus_data
+ when: monitoring_external_installation.stat.exists
+
+- name: Remove legacy Prometheus data without removing exporter textfiles
+ ansible.builtin.file:
+ path: "{{ item.path }}"
+ state: absent
+ loop: "{{ monitoring_legacy_prometheus_data.files | default([]) }}"
+ loop_control:
+ label: "{{ item.path }}"
- name: Find legacy monitoring nginx logs
ansible.builtin.find:
diff --git a/roles/monitoring/templates/nginx.conf.j2 b/roles/monitoring/templates/nginx.conf.j2
@@ -0,0 +1,47 @@
+# Managed by Ansible. Separate from the exchange webserver configuration.
+{% set node = monitoring_public_bundle.monitoring_client.node_exporter %}
+{% set bind = node.proxy_bind_address | replace('[', '') | replace(']', '') %}
+user www-data;
+worker_processes 1;
+pid /run/node-exporter-proxy.pid;
+error_log stderr warn;
+
+events {
+ worker_connections 128;
+}
+
+http {
+ access_log off;
+ server_tokens off;
+ map_hash_bucket_size 128;
+ map $ssl_client_s_dn $taler_node_exporter_client_allowed {
+ default 0;
+ {{ ('CN=' ~ node.prometheus_client_identity) | to_json }} 1;
+ }
+
+ server {
+ listen {{ '[' ~ bind ~ ']' if ':' in bind else bind }}:{{ node.proxy_port }} ssl;
+ server_name _;
+ ssl_certificate /etc/taler-monitoring/tls/server.cert.pem;
+ ssl_certificate_key /etc/taler-monitoring/tls/server.key.pem;
+ ssl_client_certificate /etc/taler-monitoring/tls/ca.cert.pem;
+ ssl_verify_client on;
+ ssl_verify_depth 1;
+ ssl_protocols TLSv1.2 TLSv1.3;
+ ssl_session_tickets off;
+
+ if ($taler_node_exporter_client_allowed = 0) {
+ return 403;
+ }
+
+ location = /metrics {
+ proxy_pass http://{{ node.backend_listen_address }};
+ proxy_http_version 1.1;
+ proxy_set_header Connection "";
+ proxy_buffering off;
+ }
+ location / {
+ return 404;
+ }
+ }
+}
diff --git a/roles/monitoring/templates/node-exporter-default.j2 b/roles/monitoring/templates/node-exporter-default.j2
@@ -0,0 +1,2 @@
+# Managed by Ansible. Plaintext metrics must remain on loopback.
+ARGS="--web.listen-address={{ monitoring_public_bundle.monitoring_client.node_exporter.backend_listen_address }} --collector.textfile.directory=/var/lib/prometheus/node-exporter --collector.systemd --collector.systemd.unit-include=.+\\.(service|socket|timer|path|target)"
diff --git a/roles/monitoring/templates/rsyslog-forward.conf.j2 b/roles/monitoring/templates/rsyslog-forward.conf.j2
@@ -0,0 +1,23 @@
+# Managed by Ansible. Forward local messages with durable RELP/TLS.
+{% set relp = monitoring_public_bundle.monitoring_client.relp %}
+module(load="omrelp" tls.tlsLib="openssl")
+
+action(
+ name="sentol_relp"
+ type="omrelp"
+ target={{ relp.server_address | to_json }}
+ port="{{ relp.server_port }}"
+ template="RSYSLOG_SyslogProtocol23Format"
+ tls="on"
+ tls.caCert="/etc/taler-monitoring/tls/ca.cert.pem"
+ tls.myCert="/etc/taler-monitoring/tls/client.cert.pem"
+ tls.myPrivKey="/etc/taler-monitoring/tls/client.key.pem"
+ tls.authMode="name"
+ tls.permittedPeer={{ relp.server_name | to_json }}
+ action.resumeRetryCount="-1"
+ queue.type="LinkedList"
+ queue.filename="sentol_relp"
+ queue.spoolDirectory="/var/spool/rsyslog"
+ queue.maxDiskSpace="1g"
+ queue.saveOnShutdown="on"
+)
diff --git a/test b/test
@@ -78,6 +78,8 @@ ansible_args=(
ansible-playbook --verbose --diff "${ansible_args[@]}" | tee "$test_state_dir/setup.log"
python3 contrib/tests/test_deployment.py \
"$test_container" "$test_state_dir/id_ed25519" "$test_state_dir/setup.log"
+python3 contrib/tests/test_monitoring.py \
+ "$test_container" "$test_state_dir/id_ed25519"
# Basic smoke checks independent of Ansible's post-deployment checks.
podman exec "$test_container" systemctl is-active --quiet \
