taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

ebisync-manual.rst (12666B)


      1 ..
      2   This file is part of GNU TALER.
      3   Copyright (C) 2025 Taler Systems SA
      4 
      5   TALER is free software; you can redistribute it and/or modify it under the
      6   terms of the GNU Affero General Public License as published by the Free Software
      7   Foundation; either version 3.0, or (at your option) any later version.
      8 
      9   TALER is distributed in the hope that it will be useful, but WITHOUT ANY
     10   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
     11   A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
     12 
     13   You should have received a copy of the GNU Affero General Public License along with
     14   TALER; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
     15 
     16   @author Antoine d'Aligny
     17 
     18 .. target audience: operator, developer
     19 
     20 .. _libeufin-ebisync:
     21 
     22 EbiSync Manual
     23 ##############
     24 
     25 LibEuFin EbiSync is an EBICS synchronization tool. It offers a command line interface to setup EBICS access and download & submit ISO20022 files.
     26 
     27 It addresses the problem that interacting with banks over EBICS requires
     28 implementing a complex cryptographic handshake to download transaction data
     29 (CAMT files) or to upload instructions for payment initiation (PAIN files).
     30 With LibEuFin EbiSync, existing systems can use LibEuFin to talk with
     31 core banking systems over EBICS without having to implement EBICS or the
     32 :ref:`Taler Wire Gateway API <taler-wire-gateway-http-api>` themselves.
     33 Instead, existing systems can simply use an HTTP POST request to submit
     34 PAIN files.  Incoming CAMT files can be automatically placed into
     35 Cloud BLOB storage and processed from there.
     36 
     37 .. image:: ../images/libeufin-ebisync.png
     38 
     39 Today, the LibEuFin implementation supports EBICS 3.0 and has been tested with
     40 Postfinance (CHF), GLS (EUR), Raiffeisen (CHF), Maerki Baumann (CHF) and Valiant (CHF). Contact
     41 us if you need support for another bank.
     42 
     43 In this manual, we explain how to setup an EBICS subscriber.  We assume that
     44 the bank has already granted EBICS access to the subscriber.
     45 
     46 Installing EbiSync
     47 ==================
     48 
     49 The following section was tested on an *OpenJDK 17* environment.
     50 
     51 
     52 Installing the libeufin-ebisync binary packages on Debian
     53 ---------------------------------------------------------
     54 
     55 .. include:: ../frags/installing-debian.rst
     56 
     57 
     58 To install libeufin-ebisync, you can now simply run:
     59 
     60 .. code-block:: console
     61 
     62    # apt install libeufin-ebisync
     63 
     64 Installing the libeufin-ebisync binary packages on Ubuntu
     65 ---------------------------------------------------------
     66 
     67 .. include:: ../frags/installing-ubuntu.rst
     68 
     69 To install libeufin-ebisync, you can now simply run:
     70 
     71 .. code-block:: console
     72 
     73    # apt install libeufin-ebisync
     74 
     75 
     76 Building from source
     77 --------------------
     78 
     79 Ebisync belongs to the LibEuFin project, and can be downloaded via Git:
     80 
     81 .. code-block:: console
     82 
     83   $ git clone git://git.taler.net/libeufin
     84 
     85 Note that Kotlin and Gradle should already work on the host system.
     86 
     87 Navigate into the *libeufin* local repository, and from top-level run:
     88 
     89 .. code-block:: console
     90 
     91   $ ./bootstrap
     92 
     93 Install deb package
     94 ~~~~~~~~~~~~~~~~~~~
     95 
     96 To install EbiSync as a Debian/Ubuntu package with an automated secure setup and systemd services:
     97 
     98 .. code-block:: console
     99 
    100   $ sudo apt install debhelper
    101   $ make deb
    102   $ sudo dpkg -i ../libeufin-ebisync*.deb
    103 
    104 If the previous steps succeeded, the ``libeufin-ebisync`` command should be found in the $PATH.
    105 
    106 
    107 Services, users, groups and file system hierarchy
    108 -------------------------------------------------
    109 
    110 The *libeufin-ebisync* package will create several system users
    111 to compartmentalize different parts of the system:
    112 
    113 * ``libeufin-ebisync``: runs all LibEuFin EbiSync components.
    114 * ``postgres``: runs the PostgreSQL database (from *postgresql* package).
    115 
    116 The package will deploy systemd service files in
    117 ``/usr/lib/systemd/system/`` for the various components:
    118 
    119 * ``libeufin-ebisync-httpd.service``: http server for the Sync API.
    120 * ``libeufin-ebisync-fetch.service``: worker daemon that retrieves files via EBICS and uploads them to a destination.
    121 * ``libeufin-ebisync.target``: main target for EbiSync to be operational.
    122 
    123 The deployment creates the following key locations in the system:
    124 
    125 * ``/etc/libeufin-ebisync/``: configuration files.
    126 * ``/run/libeufin-ebisync/``: contains the UNIX domain sockets for inter-process communication (IPC).
    127 * ``/var/lib/libeufin-ebisync/``: serves as the $HOME for libeufin-ebisync and contains the EBICS authentication keys.
    128 
    129 Database setup
    130 ==============
    131 
    132 The configuration file must include a connection string that
    133 tells EbiSync how it should connect to the database. The default
    134 is:
    135 
    136 .. code-block:: ini
    137    :caption: /etc/libeufin-ebisync/secrets/ebisync-db.secret.conf
    138 
    139    [ebisyncdb-postgres]
    140    config = postgres:///libeufin-ebisync
    141 
    142 For a local PostgreSQL installation, the following helper creates the database
    143 and role, initializes (or upgrades) the schema, and grants service access:
    144 
    145 .. code-block:: console
    146 
    147   $ sudo libeufin-ebisync-dbconfig
    148 
    149 EBICS setup
    150 ===========
    151 
    152 When you sign up for an EBICS-enabled bank account, the bank will provide you
    153 with various credentials. Those must be provided in the
    154 ``/etc/libeufin-ebisync/libeufin-ebisync.conf`` configuration file.
    155 
    156 The following snippet shows the mandatory configuration values:
    157 
    158 .. _ebisync-core-config:
    159 
    160 .. code-block:: ini
    161   :caption: /etc/libeufin-ebisync/libeufin-ebisync.conf
    162 
    163   [ebisync]
    164   # Bank
    165   HOST_BASE_URL = https://ebics.postfinance.ch/ebics/ebics.aspx
    166 
    167   # EBICS IDs
    168   HOST_ID = PFEBICS
    169   USER_ID = PFC00563
    170   PARTNER_ID = PFC00563
    171 
    172 .. note::
    173   Refer to the manpage ``libeufin-ebisync.conf(5)``
    174   for the full array of configuration values.
    175 
    176 .. warning::
    177   This combination of HOST_ID, USER_ID and PARTNER_ID must never be used by another instance of libeufin-ebisync or by other EBICS clients, otherwise data will be lost.
    178 
    179 Reuse existing client keys
    180 --------------------------
    181 
    182 If you have client keys from a previous EBICS setup you can copy the JSON file to the configured path ``CLIENT_PRIVATE_KEYS_FILE`` (``/var/lib/libeufin-ebisync/client-ebics-keys.json`` with the default config).
    183 
    184 Make sure this file is accessible to the user running ``libeufin-ebisync``, for the default services you should run:
    185 
    186 .. code-block:: console
    187 
    188   $ chown libeufin-ebisync:libeufin-ebisync /var/lib/libeufin-ebisync/client-ebics-keys.json
    189 
    190 Create new client keys
    191 ----------------------
    192 
    193 The following command should start the EBICS setup process:
    194 
    195 .. code-block:: console
    196 
    197   $ sudo -u libeufin-ebisync libeufin-ebisync setup -c /etc/libeufin-ebisync/libeufin-ebisync.conf
    198 
    199 If the previous command failed when running EBICS INI with an error code of
    200 ``EBICS_INVALID_USER_OR_USER_STATE``, you need to confirm your keys to your bank to
    201 activate your account.
    202 
    203 To that end, the previous run should have left a PDF document that you can
    204 print, sign and send to the bank.  Look for the message that looks like ``PDF
    205 file with keys created at '/tmp/libeufin-ebics-keys/$TIMESTAMP.pdf'``.
    206 
    207 Once the bank has received and processed this document you can continue.
    208 
    209 Get bank keys
    210 -------------
    211 
    212 The following command will finish the EBICS setup process:
    213 
    214 .. code-block:: console
    215 
    216   $ sudo -u libeufin-ebisync libeufin-ebisync setup -c /etc/libeufin-ebisync/libeufin-ebisync.conf
    217 
    218 The EBICS setup is finished once the bank keys have been accepted.
    219 
    220 Fetch destination setup
    221 =======================
    222 
    223 Azure Blob Storage
    224 ------------------
    225 
    226 The following snippet shows the mandatory configuration values:
    227 
    228 .. code-block:: ini
    229   :caption: /etc/libeufin-ebisync/conf.d/ebisync-fetch.conf
    230 
    231   [ebisync-fetch]
    232   DESTINATION = azure-blob-storage
    233 
    234 Some configuration values are sensitive and must be written to a secret configuration file that is only accessible to the ``libeufin-ebisync`` user.
    235 
    236 .. code-block:: ini
    237   :caption: /etc/libeufin-ebisync/secrets/ebisync-fetch.secret.conf
    238 
    239   [ebisync-fetch]
    240   AZURE_API_URL = https://myaccount.blob.core.windows.net/
    241   AZURE_ACCOUNT_NAME = myaccount
    242   AZURE_ACCOUNT_KEY = Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
    243   AZURE_CONTAINER = mycontainer
    244 
    245 The following command will finish the fetcher setup process:
    246 
    247 .. code-block:: console
    248 
    249   $ sudo -u libeufin-ebisync libeufin-ebisync setup -c /etc/libeufin-ebisync/libeufin-ebisync.conf
    250 
    251 Submit source setup
    252 ===================
    253 
    254 EbiSync API
    255 -----------
    256 
    257 The following snippet shows the mandatory configuration values:
    258 
    259 .. code-block:: ini
    260   :caption: /etc/libeufin-ebisync/conf.d/ebisync-submit.conf
    261 
    262   [ebisync-submit]
    263   SOURCE = ebisync-api
    264 
    265 Some configuration values are sensitive and must be written to a secret configuration file that is only accessible to the ``libeufin-ebisync`` user.
    266 
    267 .. code-block:: ini
    268   :caption: /etc/libeufin-ebisync/secrets/ebisync-submit.secret.conf
    269 
    270   [ebisync-submit]
    271   AUTH_METHOD = basic
    272   USERNAME = admin
    273   PASSWORD = password
    274 
    275 Check the server configuration:
    276 
    277 .. code-block:: console
    278 
    279   $ sudo -u libeufin-ebisync libeufin-ebisync serve --check -c /etc/libeufin-ebisync/libeufin-ebisync.conf
    280 
    281 Fetching files
    282 ==============
    283 
    284 The responsible subcommand for fetching and uploading files is
    285 ``fetch``, and its configuration is a **superset** of ebisync-core-config_. On
    286 top of that, it expects the database connection string, the destination config and *optionally* a
    287 frequency and a checkpoint time of day.
    288 
    289 ``fetch`` will run a periodical fetch of pending documents (documents
    290 that have never been fetched before). If the bank server supports it, it will
    291 also listen to real-time document availability notifications and fetch
    292 documents as soon as they are announced. This enables instant registration of
    293 instant transactions. If your bank server does not support real-time
    294 notifications, it is recommended that you fetch more often; otherwise, you can
    295 reduce the fetch frequency.
    296 
    297 Only fetching pending documents is not always reliable, if other EBICS clients
    298 use the same connection settings as ``libeufin-ebisync``, they can consume documents before they are ingested. In case of bank downtime, the status of
    299 documents can be lost. This is why ``fetch`` will run a checkpoint every
    300 day to download all documents generated since the last checkpoint.
    301 
    302 If the bank supports real-time notifications, you can expect transactions to be registered as soon as they are booked: ~10s for instant transactions and 24 to 72h for other transactions. Otherwise, expect a latency corresponding to the fetch frequency for instant transactions. Thanks to checkpointing, latency should not exceed one day.
    303 
    304 The frequency **may** be specified as
    305 
    306 .. code-block:: ini
    307   :caption: /etc/libeufin-ebisync/conf.d/ebisync-fetch.conf
    308 
    309   [ebisync-fetch]
    310   FREQUENCY = 5m
    311 
    312 The supported time units are ``s`` (seconds), ``m`` (minutes), ``h`` (hours).
    313 
    314 The checkpoint time of day **may** be specified as
    315 
    316 .. code-block:: ini
    317   :caption: /etc/libeufin-ebisync/conf.d/ebisync-fetch.conf
    318 
    319   [ebisync-fetch]
    320   CHECKPOINT_TIME_OF_DAY = 19:00
    321 
    322 The following command would download any unseen EBICS files:
    323 
    324 .. code-block:: console
    325 
    326   $ sudo -u libeufin-ebisync libeufin-ebisync fetch -c /etc/libeufin-ebisync/libeufin-ebisync.conf --transient
    327 
    328 The ``fetch`` subcommand will always cause libeufin-ebisync to download EBICS files and upload them to the configured destination.
    329 
    330 The ``--transient`` flag makes the command sync only
    331 once and return immediately afterwards.  Without the flag, EbiSync
    332 will download at the frequency specified in the configuration or when it receives real-time notifications.
    333 
    334 The ``--debug-ebics $SAVEDIR`` flag will cause EBICS transactions steps and payloads to be stored in ``$SAVEDIR/$YYYY-MM-DD`` where ``$YYYY-MM-DD`` represents the date when the download took place. This is mostly useful for debugging.
    335 
    336 Submitting files
    337 =================
    338 
    339 If the ``ebisync-api`` source is configured, submission is triggered by using the API.
    340 
    341 Deployment
    342 ==========
    343 
    344 Reverse Proxy Setup
    345 -------------------
    346 
    347 By default if ``ebisync-api`` is used as a source, the ``libeufin-ebisync-httpd`` service listens for HTTP connections
    348 on localhost.  To make the service publicly available, a reverse
    349 proxy such as nginx should be used.  We strongly recommend to configure nginx
    350 to use TLS.
    351 
    352 The ``libeufin-ebisync`` package ships with a sample configuration that can be
    353 enabled in nginx:
    354 
    355 .. code-block:: shell-session
    356 
    357   # vim /etc/nginx/sites-available/libeufin-ebisync
    358   < ... customize configuration ... >
    359   # ln -s /etc/nginx/sites-available/libeufin-ebisync /etc/nginx/sites-enabled/libeufin-ebisync
    360   # systemctl reload nginx
    361 
    362 Launching
    363 ---------
    364 
    365 .. code-block:: console
    366 
    367   $ sudo systemctl enable --now libeufin-ebisync.target