taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

sakana.rst (4482B)


      1 ..
      2   This file is part of GNU TALER.
      3   Copyright (C) 2024, 2026 Taler Systems SA
      4 
      5   TALER is free software; you can redistribute it and/or modify it under the
      6   terms of the GNU Affero General Public License as published by the Free Software
      7   Foundation; either version 3.0, or (at your option) any later version.
      8 
      9   TALER is distributed in the hope that it will be useful, but WITHOUT ANY
     10   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
     11   A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
     12 
     13   You should have received a copy of the GNU Affero General Public License along with
     14   TALER; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
     15 
     16   @author Christian Grothoff
     17 
     18 Sakana
     19 ######
     20 
     21 Sakana runs alongside :ref:`taler-monitoring`. It checks GNU Taler exchanges
     22 for problems that an availability probe does not see, and it checks Sentol's
     23 own Prometheus and Alertmanager from the outside: it is the monitor that
     24 monitors the monitoring. Its checks and its reports are configured separately
     25 from Sentol's Prometheus probes and Alertmanager routing.
     26 
     27 The source is at ``git://git.taler.net/sakana.git``, and the ``sakana``
     28 package is on deb.taler.net. ``sakana(1)`` documents the command,
     29 ``sakana.conf(5)`` every configuration option, and ``info sakana`` is the
     30 full manual.
     31 
     32 
     33 Configuration
     34 =============
     35 
     36 Sakana reads ``/etc/sakana/sakana.conf``. Every section other than
     37 ``[sakana]`` describes one monitored system; ``KIND`` selects ``exchange``
     38 (the default), ``prometheus`` or ``alertmanager``, and ``BASE_URL`` is the
     39 system's base URL, not an endpoint. The ``[sakana]`` section holds defaults
     40 for all systems, notably:
     41 
     42 - ``FREQUENCY``: how often each system is checked;
     43 - ``TOLERANCE``: how many consecutive failures of one error class are
     44   tolerated before a report is made;
     45 - ``COMMAND``: the command that receives each report on its standard input.
     46   The first line of a report is a summary suitable as a mail subject, and
     47   ``/usr/share/sakana/sakana-notify-mail`` delivers reports by email.
     48 
     49 Each error class of each system has its own tolerance, so a long-standing
     50 warning never delays the report of a new outage. Once a system passes every
     51 check again, Sakana sends a brief "service restored" report.
     52 
     53 The file may hold credentials (``AUTH_BASIC``, ``AUTH_BEARER``, ``ENV_*``),
     54 so keep it mode 0640, owned by ``root:sakana``. Run ``sakana --test`` to
     55 check a configuration without contacting anything.
     56 
     57 
     58 Monitoring exchanges
     59 ====================
     60 
     61 To monitor an exchange, we want to not merely monitor that the REST API is
     62 operational, but also check that the ``/keys`` response is not outdated.
     63 Besides failures to connect, failed TLS handshakes, HTTP errors and timeouts,
     64 Sakana reports:
     65 
     66 - a TLS certificate that expires within 5 days (``TLS_RENEWAL_HORIZON``),
     67   by when certbot should have renewed it;
     68 - signing keys of which none is valid beyond the next 30 days
     69   (``SIGNKEY_EXPIRATION_HORIZON``);
     70 - denominations of which none can be withdrawn beyond the next 14 days
     71   (``DENOMINATION_EXPIRATION_HORIZON``). This horizon has to stay below the
     72   ``LOOKAHEAD_SIGN`` of the monitored exchange, or a correctly rotating
     73   exchange warns on every check;
     74 - global fees or wire fees that are not configured for the current year
     75   (and, in December, for the next year).
     76 
     77 Exchanges to monitor include:
     78 
     79 - exchange.taler-ops.ch
     80 - exchange.e.netzbon-basel.ch
     81 - exchange.demo.taler.net
     82 - exchange.head.taler.net
     83 - exchange.test.taler.net
     84 
     85 
     86 Monitoring the monitoring
     87 =========================
     88 
     89 When an Alertmanager stops working, the symptom is silence, and silence
     90 looks the same as everything being in order. Sakana therefore also checks
     91 Sentol's Prometheus (``KIND = prometheus``) and Alertmanager
     92 (``KIND = alertmanager``) and reports through its own ``COMMAND``, not
     93 through Alertmanager.
     94 
     95 For both, Sakana reports an instance that is running but not ready, and one
     96 that restarted since the previous check. For Prometheus, it also reports a
     97 failed configuration reload, a corrupted time series database and rules that
     98 cannot be evaluated. For Alertmanager, it also reports a gossip cluster with
     99 fewer peers than ``CLUSTER_PEERS`` requires.
    100 
    101 ``BASE_URL`` is the prefix that the service's ``--web.external-url`` names,
    102 since that is where its ``/-/healthy`` endpoint lives. Sentol's web interfaces
    103 require a password (see :ref:`taler-monitoring`), so these sections also need
    104 ``AUTH_BASIC``.