commit 0e0979803e47dc439d41128c5a04f39435b672c4
parent e279a25441823d074d28ee5586e1821cf3aebdae
Author: Christian Grothoff <christian@grothoff.org>
Date: Sun, 15 Nov 2026 21:16:53 +0100
uptimekuma to sakana
Diffstat:
4 files changed, 110 insertions(+), 152 deletions(-)
diff --git a/system-administration/index.rst b/system-administration/index.rst
@@ -23,7 +23,7 @@ Internal System Administration
:glob:
taler-monitoring-infrastructure
- uptime-kuma
+ sakana
backups
support
debian-repos
diff --git a/system-administration/sakana.rst b/system-administration/sakana.rst
@@ -0,0 +1,104 @@
+..
+ This file is part of GNU TALER.
+ Copyright (C) 2024, 2026 Taler Systems SA
+
+ TALER is free software; you can redistribute it and/or modify it under the
+ terms of the GNU Affero General Public License as published by the Free Software
+ Foundation; either version 3.0, or (at your option) any later version.
+
+ TALER is distributed in the hope that it will be useful, but WITHOUT ANY
+ WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+ A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
+
+ You should have received a copy of the GNU Affero General Public License along with
+ TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/>
+
+ @author Christian Grothoff
+
+Sakana
+######
+
+Sakana runs alongside :ref:`taler-monitoring`. It checks GNU Taler exchanges
+for problems that an availability probe does not see, and it checks Sentol's
+own Prometheus and Alertmanager from the outside: it is the monitor that
+monitors the monitoring. Its checks and its reports are configured separately
+from Sentol's Prometheus probes and Alertmanager routing.
+
+The source is at ``git://git.taler.net/sakana.git``, and the ``sakana``
+package is on deb.taler.net. ``sakana(1)`` documents the command,
+``sakana.conf(5)`` every configuration option, and ``info sakana`` is the
+full manual.
+
+
+Configuration
+=============
+
+Sakana reads ``/etc/sakana/sakana.conf``. Every section other than
+``[sakana]`` describes one monitored system; ``KIND`` selects ``exchange``
+(the default), ``prometheus`` or ``alertmanager``, and ``BASE_URL`` is the
+system's base URL, not an endpoint. The ``[sakana]`` section holds defaults
+for all systems, notably:
+
+- ``FREQUENCY``: how often each system is checked;
+- ``TOLERANCE``: how many consecutive failures of one error class are
+ tolerated before a report is made;
+- ``COMMAND``: the command that receives each report on its standard input.
+ The first line of a report is a summary suitable as a mail subject, and
+ ``/usr/share/sakana/sakana-notify-mail`` delivers reports by email.
+
+Each error class of each system has its own tolerance, so a long-standing
+warning never delays the report of a new outage. Once a system passes every
+check again, Sakana sends a brief "service restored" report.
+
+The file may hold credentials (``AUTH_BASIC``, ``AUTH_BEARER``, ``ENV_*``),
+so keep it mode 0640, owned by ``root:sakana``. Run ``sakana --test`` to
+check a configuration without contacting anything.
+
+
+Monitoring exchanges
+====================
+
+To monitor an exchange, we want to not merely monitor that the REST API is
+operational, but also check that the ``/keys`` response is not outdated.
+Besides failures to connect, failed TLS handshakes, HTTP errors and timeouts,
+Sakana reports:
+
+- a TLS certificate that expires within 5 days (``TLS_RENEWAL_HORIZON``),
+ by when certbot should have renewed it;
+- signing keys of which none is valid beyond the next 30 days
+ (``SIGNKEY_EXPIRATION_HORIZON``);
+- denominations of which none can be withdrawn beyond the next 14 days
+ (``DENOMINATION_EXPIRATION_HORIZON``). This horizon has to stay below the
+ ``LOOKAHEAD_SIGN`` of the monitored exchange, or a correctly rotating
+ exchange warns on every check;
+- global fees or wire fees that are not configured for the current year
+ (and, in December, for the next year).
+
+Exchanges to monitor include:
+
+- exchange.taler-ops.ch
+- exchange.e.netzbon-basel.ch
+- exchange.demo.taler.net
+- exchange.head.taler.net
+- exchange.test.taler.net
+
+
+Monitoring the monitoring
+=========================
+
+When an Alertmanager stops working, the symptom is silence, and silence
+looks the same as everything being in order. Sakana therefore also checks
+Sentol's Prometheus (``KIND = prometheus``) and Alertmanager
+(``KIND = alertmanager``) and reports through its own ``COMMAND``, not
+through Alertmanager.
+
+For both, Sakana reports an instance that is running but not ready, and one
+that restarted since the previous check. For Prometheus, it also reports a
+failed configuration reload, a corrupted time series database and rules that
+cannot be evaluated. For Alertmanager, it also reports a gossip cluster with
+fewer peers than ``CLUSTER_PEERS`` requires.
+
+``BASE_URL`` is the prefix that the service's ``--web.external-url`` names,
+since that is where its ``/-/healthy`` endpoint lives. Sentol's web interfaces
+require a password (see :ref:`taler-monitoring`), so these sections also need
+``AUTH_BASIC``.
diff --git a/system-administration/taler-monitoring-infrastructure.rst b/system-administration/taler-monitoring-infrastructure.rst
@@ -49,7 +49,7 @@ The stack runs as native systemd services on Sentol and the monitored hosts:
.. mermaid::
:config: {"layout": "elk"}
:caption: Monitoring architecture; arrows show requests or log and alert delivery.
- :alt: Prometheus scrapes host metrics over mTLS and service health through exporters. Fluent Bit sends journals over mTLS to VictoriaLogs. Perses provides dashboards, Alertmanager sends email, and nginx provides authenticated web access.
+ :alt: Prometheus scrapes host metrics over mTLS and service health through exporters. Fluent Bit sends journals over mTLS to VictoriaLogs. Perses provides dashboards, Alertmanager sends email, and nginx provides authenticated web access. Sakana checks exchange keys and, through nginx, Prometheus and Alertmanager.
flowchart TB
subgraph host["Monitored host"]
@@ -83,7 +83,8 @@ The stack runs as native systemd services on Sentol and the monitored hosts:
probes -->|"HTTP(S) and /config"| endpoints["Public services"]
browser["Operator browser"] -->|"HTTPS / password"| web
alert --> smtp["SMTP relay"]
- kuma["Uptime Kuma"] -->|"independent checks"| endpoints
+ sakana["Sakana"] -->|"exchange /keys"| endpoints
+ sakana -->|"Prometheus and Alertmanager health"| web
Sentol also monitors itself. Collector and server backends listen on loopback;
nginx exposes the authenticated interfaces. One private monitoring CA secures
@@ -93,8 +94,8 @@ Hosts are either managed directly by ``tsys-infra.git`` or enrolled as external
clients. ``ansible-taler-exchange`` uses the latter model: it configures exchange
hosts from bundles exported by Sentol.
-:doc:`uptime-kuma` provides independent availability monitoring, including
-exchange key and fee checks. Its checks and notifications are configured
+:doc:`sakana` checks exchange keys and fees, and checks Sentol's Prometheus
+and Alertmanager from the outside. Its checks and reports are configured
separately from Sentol.
Web interfaces
diff --git a/system-administration/uptime-kuma.rst b/system-administration/uptime-kuma.rst
@@ -1,147 +0,0 @@
-..
- This file is part of GNU TALER.
- Copyright (C) 2024 Taler Systems SA
-
- TALER is free software; you can redistribute it and/or modify it under the
- terms of the GNU Affero General Public License as published by the Free Software
- Foundation; either version 3.0, or (at your option) any later version.
-
- TALER is distributed in the hope that it will be useful, but WITHOUT ANY
- WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
- A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License along with
- TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/>
-
- @author Christian Grothoff
-
-Uptime Kuma
-###########
-
-Uptime Kuma remains in use alongside :ref:`taler-monitoring`. It provides
-independent public availability checks and the exchange key and fee checks
-described below. Its monitors and notifications are configured separately from
-Sentol's Prometheus probes and Alertmanager routing.
-
-We use https://github.com/louislam/uptime-kuma to monitor public server and service
-availability. We run two servers:
-
-- https://uptimekuma.anastasis.lu/ monitors almost everything, and is our production
- monitor hosted by Anastasis SARL.
-
-- https://uptimekuma.taler.net/ is our secondary monitor, which monitors the
- production monitor.
-
-
-Monitoring servers
-==================
-
-We monitor server availability using "PING":
-
-- gv.taler.net
-- firefly.gnunet.org
-- pixel.anastasis.lu
-
-Monitoring websites
-===================
-
-For most websites, we simply use the HTTP(s) monitor. Make sure to enable advance
-notice for TLS certificate expiration. Sites to monitor include:
-
-- anastasis.lu
-- docs.anastasis.lu
-- webui.anastasis.lu
-- gnunet.org
-- bib.gnunet.org
-- bugs.gnunet.org
-- docs.gnunet.org
-- gana.gnunet.org
-- git.gnunet.org
-- grafana.gnunet.org
-- lsd.gnunet.org
-- taler.net
-- bugs.taler.net
-- demo.taler.net
-- docs.taler.net
-- git.taler.net
-- mattermost.taler.net
-- sandstorm.taler.net
-- test.taler.net
-- tutorials.taler.net
-- weblate.taler.net
-- taler-ops.ch
-- taler-systems.com
-- schanzen.eu
-
-Monitoring REST APIs
-====================
-
-For simple REST APIs, we simply use the HTTP(s) monitor on the /config endpoint.
-Make sure to enable advance notice for TLS certificate expiration. APIs to monitor include:
-
-- v1.anastasis.lu
-- backend.demo.taler.net
-- backend.head.taler.net
-- backend.test.taler.net
-
-
-Monitoring services
-===================
-
-We also monitor the following services using the HTTP(s) monitor on the "/" endpoint:
-
-- davical.taler.net
-- fcfs.gnunet.org
-
-
-Monitoring exchanges
-====================
-
-To monitor an exchange, we want to not merely monitor that the REST API is operational,
-but also check that the ``/keys`` response is not outdated. This can be done by computing
-the minimum expiration time over all of the maxima of the different keys in the response.
-The HTTP(S) - JSON Query method can be
-used using the following query written in https://jsonata.org/:
-
-.. code-block:: text
-
- $min($append(
- [$max($map(signkeys, function($f) { $f.stamp_expire.t_s}))],
- [$max($map(denominations,
- function ($a){
- $max($map($a.denoms,
- function ($d){
- $d.stamp_expire_withdraw.t_s
- }
- ))
- }
- ))])) > $millis()/1000
-
-We setup a separate monitor to check that wire fees are working
-(sure, they can be combined, but this way we can get a more precise alert):
-
-.. code-block:: none
-
- $max(wire_fees[*].[*].end_date.t_s) > $millis()/1000
-
-To separate-out global fees monitoring (which is not present on all systems),
-we also use (except on GLS, which doesn't support P2P):
-
-.. code-block:: none
-
- $max(global_fees[*].end_date.t_s) > $millis()/1000
-
-Exchanges to monitor include:
-
-- exchange.taler-ops.ch
-- exchange.e.netzbon-basel.ch
-- exchange.demo.taler.net
-- exchange.head.taler.net
-- exchange.test.taler.net
-
-
-Tagging monitors
-================
-
-We tag each monitor with the host that the service
-is running on (e.g. firefly.gnunet.org).