sakana.rst (4482B)
1 .. 2 This file is part of GNU TALER. 3 Copyright (C) 2024, 2026 Taler Systems SA 4 5 TALER is free software; you can redistribute it and/or modify it under the 6 terms of the GNU Affero General Public License as published by the Free Software 7 Foundation; either version 3.0, or (at your option) any later version. 8 9 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 10 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 11 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 12 13 You should have received a copy of the GNU Affero General Public License along with 14 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 15 16 @author Christian Grothoff 17 18 Sakana 19 ###### 20 21 Sakana runs alongside :ref:`taler-monitoring`. It checks GNU Taler exchanges 22 for problems that an availability probe does not see, and it checks Sentol's 23 own Prometheus and Alertmanager from the outside: it is the monitor that 24 monitors the monitoring. Its checks and its reports are configured separately 25 from Sentol's Prometheus probes and Alertmanager routing. 26 27 The source is at ``git://git.taler.net/sakana.git``, and the ``sakana`` 28 package is on deb.taler.net. ``sakana(1)`` documents the command, 29 ``sakana.conf(5)`` every configuration option, and ``info sakana`` is the 30 full manual. 31 32 33 Configuration 34 ============= 35 36 Sakana reads ``/etc/sakana/sakana.conf``. Every section other than 37 ``[sakana]`` describes one monitored system; ``KIND`` selects ``exchange`` 38 (the default), ``prometheus`` or ``alertmanager``, and ``BASE_URL`` is the 39 system's base URL, not an endpoint. The ``[sakana]`` section holds defaults 40 for all systems, notably: 41 42 - ``FREQUENCY``: how often each system is checked; 43 - ``TOLERANCE``: how many consecutive failures of one error class are 44 tolerated before a report is made; 45 - ``COMMAND``: the command that receives each report on its standard input. 46 The first line of a report is a summary suitable as a mail subject, and 47 ``/usr/share/sakana/sakana-notify-mail`` delivers reports by email. 48 49 Each error class of each system has its own tolerance, so a long-standing 50 warning never delays the report of a new outage. Once a system passes every 51 check again, Sakana sends a brief "service restored" report. 52 53 The file may hold credentials (``AUTH_BASIC``, ``AUTH_BEARER``, ``ENV_*``), 54 so keep it mode 0640, owned by ``root:sakana``. Run ``sakana --test`` to 55 check a configuration without contacting anything. 56 57 58 Monitoring exchanges 59 ==================== 60 61 To monitor an exchange, we want to not merely monitor that the REST API is 62 operational, but also check that the ``/keys`` response is not outdated. 63 Besides failures to connect, failed TLS handshakes, HTTP errors and timeouts, 64 Sakana reports: 65 66 - a TLS certificate that expires within 5 days (``TLS_RENEWAL_HORIZON``), 67 by when certbot should have renewed it; 68 - signing keys of which none is valid beyond the next 30 days 69 (``SIGNKEY_EXPIRATION_HORIZON``); 70 - denominations of which none can be withdrawn beyond the next 14 days 71 (``DENOMINATION_EXPIRATION_HORIZON``). This horizon has to stay below the 72 ``LOOKAHEAD_SIGN`` of the monitored exchange, or a correctly rotating 73 exchange warns on every check; 74 - global fees or wire fees that are not configured for the current year 75 (and, in December, for the next year). 76 77 Exchanges to monitor include: 78 79 - exchange.taler-ops.ch 80 - exchange.e.netzbon-basel.ch 81 - exchange.demo.taler.net 82 - exchange.head.taler.net 83 - exchange.test.taler.net 84 85 86 Monitoring the monitoring 87 ========================= 88 89 When an Alertmanager stops working, the symptom is silence, and silence 90 looks the same as everything being in order. Sakana therefore also checks 91 Sentol's Prometheus (``KIND = prometheus``) and Alertmanager 92 (``KIND = alertmanager``) and reports through its own ``COMMAND``, not 93 through Alertmanager. 94 95 For both, Sakana reports an instance that is running but not ready, and one 96 that restarted since the previous check. For Prometheus, it also reports a 97 failed configuration reload, a corrupted time series database and rules that 98 cannot be evaluated. For Alertmanager, it also reports a gossip cluster with 99 fewer peers than ``CLUSTER_PEERS`` requires. 100 101 ``BASE_URL`` is the prefix that the service's ``--web.external-url`` names, 102 since that is where its ``/-/healthy`` endpoint lives. Sentol's web interfaces 103 require a password (see :ref:`taler-monitoring`), so these sections also need 104 ``AUTH_BASIC``.