commit f191768a1821475eae90db78a4efe03f87ec06b5
parent 7bb4aba1f97dd15c1d1286394d8db9add557da1d
Author: Florian Dold <dold@taler.net>
Date: Sat, 5 Sep 2026 18:55:06 +0200
DD 102: define socket lifetime and dependency recovery
Bind activation sockets to their service so permanent failures and explicit
stops prevent traffic from starting the daemon again. Request reconnectable
dependencies through shared product targets so consumer restarts leave
stopped dependencies alone.
Define the recovery boundary for systemd prerequisite failures and require
monitoring for unexpectedly inactive services and failed prerequisites.
Diffstat:
1 file changed, 190 insertions(+), 22 deletions(-)
diff --git a/design-documents/102-systemd-service-restart-policy.rst b/design-documents/102-systemd-service-restart-policy.rst
@@ -6,19 +6,26 @@ DD 102: systemd service restart policy
:DD shepherd: Florian Dold
:Historical contributors: Florian Dold
:First published: 2026-09-03
-:Last substantive change: 2026-09-03
+:Last substantive change: 2026-09-05
Summary
========
-Long-running Taler services should recover from transient failures without
-operator intervention. They should stop on invalid configuration or another
-known permanent failure, and repeated crashes must be visible to operators.
+Long-running Taler services should recover from transient daemon failures
+without operator intervention. They should stop on invalid configuration or
+another known permanent failure, and repeated crashes must be visible to
+operators.
The proposed policy retries indefinitely with a fixed ten-second delay. Exit
status 6 means that the service is not configured correctly; exit status 9
means that the service encountered another permanent failure. Neither status
-is restarted.
+causes an automatic restart. Associated activation sockets stop with the
+service so that incoming traffic cannot start it again after a permanent
+failure or an explicit stop.
+
+This recovery guarantee assumes that systemd can start the daemon. Failures
+in systemd prerequisites, such as a socket that cannot bind, require monitoring
+and operator repair.
Systemd 257 in Debian Trixie supports an increasing restart delay, but does
not reset that delay after the service has been healthy for a long time. If a
@@ -45,6 +52,12 @@ status 6 for configuration errors and status 9 for failures that should not be
retried, but not all units recognize both statuses and non-C services do not
always return them.
+Socket activation and unit dependencies also affect recovery. Incoming
+traffic can start a service again after ``RestartPreventExitStatus=`` has
+suppressed its automatic restart, without observing ``RestartSec=``. In the
+other direction, a failed dependency with ``Requires=`` and ``After=`` can
+prevent the daemon from starting at all, leaving its restart policy unused.
+
An unlimited restart policy also needs monitoring. In particular,
``RestartMode=direct`` skips the failed/inactive transition during automatic
restarts and does not invoke ``OnFailure=`` units. A service can therefore
@@ -54,12 +67,13 @@ state.
Requirements
============
-* Long-running services keep retrying after transient failures.
+* Long-running services keep retrying after transient daemon failures,
+ including temporary loss of reconnectable dependencies.
* Restarts do not form a tight loop.
* Invalid configuration and known permanent failures are not retried.
* Exit statuses have the same meaning in every implementation language.
-* An explicit ``systemctl stop`` still stops the service.
-* Operators can detect stopped services and crash loops.
+* An explicit ``systemctl stop`` stops the service and its activation sockets.
+* Operators can detect stopped services, failed prerequisites, and crash loops.
* The policy works with Debian Trixie.
Proposed Solution
@@ -71,9 +85,19 @@ Scope
The policy applies to long-running product services shipped or deployed by
Taler, Anastasis, LibEuFin, Donau, Paivana, Challenger, and related
repositories, including distribution-specific copies of their units.
+The socket-lifetime rules below also apply to their associated activation
+sockets.
+
+The automatic recovery guarantee covers failures handled by the daemon and
+its service restart policy. It does not cover failures in systemd
+prerequisites, such as failed assertions, socket bind failures, or dependency
+start-job failures. These may leave a service inactive without ever running
+its main process. Deployments must detect these cases and provide operator
+recovery; disabling the service start-rate limiter does not resolve them.
-It does not apply to one-shot initialization commands, timer-triggered jobs,
-garbage collection jobs, or other processes that are expected to finish.
+The service baseline does not apply to one-shot initialization commands,
+timer-triggered jobs, garbage collection jobs, or other processes that are
+expected to finish.
Unit policy
-----------
@@ -90,8 +114,8 @@ Long-running services use this baseline:
RestartSec=10s
RestartPreventExitStatus=6 9
-``StartLimitIntervalSec=0`` disables the start-rate limiter. This is needed
-because ``Restart=`` remains subject to start-rate limiting.
+``StartLimitIntervalSec=0`` disables the service's start-rate limiter. This
+is needed because ``Restart=`` remains subject to start-rate limiting.
``Restart=always`` restarts the service after clean and unsuccessful exits,
signals, timeouts, and watchdog failures. It does not override an explicit
@@ -103,6 +127,94 @@ unrelated failures. Units must not add ``StartLimitBurst``, a non-zero
``RestartSteps``, or ``RestartMaxDelaySec``. A unit may prevent additional
exit statuses when its program documents them as permanent failures.
+Socket lifetime
+---------------
+
+For a service with activation sockets, apply the baseline above and add the
+following to ``example.service``:
+
+.. code-block:: ini
+
+ [Unit]
+ Requires=example.socket
+
+ [Service]
+ RestartMode=direct
+
+Its ``example.socket`` uses this lifetime relationship, in addition to its
+existing listen address and access settings:
+
+.. code-block:: ini
+
+ [Unit]
+ BindsTo=example.service
+
+ [Socket]
+ Accept=no
+ Service=example.service
+
+Apply this relationship to every activation socket of the service, including
+the corresponding instances of template units. The service requires all
+sockets whose file descriptors it needs. Do not add ``After=example.service``
+to a socket: systemd already orders activation sockets before their services,
+so the opposite ordering would introduce a cycle.
+
+``BindsTo=`` stops the sockets when the service stops or fails permanently.
+``RestartMode=direct`` is required for this profile: it avoids the transient
+failed/inactive state during automatic restarts, so those restarts keep the
+sockets available. Pending connections wait for the next restart; they must
+not shorten the ten-second delay. ``PartOf=`` alone does not provide the
+required behavior on permanent failure.
+
+Starting a socket with this relationship also starts the daemon. These are
+long-running services, rather than daemons started only when traffic arrives.
+After exit status 6 or 9, the service remains failed and its sockets stop.
+An explicit ``systemctl stop example.service`` also stops the sockets, so
+subsequent traffic cannot reactivate the service. After repairing the failure,
+``systemctl start example.service`` starts the service and its required sockets.
+
+``RestartPreventExitStatus=`` only suppresses automatic service restarts. It
+does not reject independent start requests, including socket activation or an
+operator's explicit start. Deployments must not add unconditional activation
+mechanisms, such as ``Upholds=`` or a handler that always starts the service,
+where they would override permanent exits or explicit stops. Socket trigger
+limits are separate from the service start-rate limiter and are not a
+substitute for the lifetime relationship above.
+
+Reconnectable dependencies
+--------------------------
+
+For dependencies that the daemon can reconnect to, such as a database or
+security module, request local startup through a shared product target and
+order startup with ``After=`` on the consumer. For example,
+``example.target`` uses ``Wants=example.service example-dependency.service``,
+while ``example.service`` uses ``After=example-dependency.service``. A failed
+dependency then does not prevent the consumer from starting.
+
+Replace ``Requires=`` or ``BindsTo=`` relationships from the consumer that
+would block its startup or stop it when a reconnectable dependency goes away.
+A failed start job with ``Requires=`` and ``After=`` does not schedule an
+automatic restart of the consumer; starting the dependency later does not
+automatically start that service either.
+
+Do not simply replace these relationships with ``Wants=`` on the consumer.
+An automatic restart of the consumer also pulls in its wanted units, which
+can start a dependency again after a permanent exit or explicit stop. With
+``Wants=`` on the shared target instead, consumer restarts do not reactivate
+the dependency. Start the product target to request startup of the group;
+starting the consumer alone does not start its reconnectable dependencies.
+
+The daemon must handle an unavailable dependency by reconnecting internally
+or returning a restartable status, such as 1. Temporary unavailability must
+not be classified as invalid configuration or a permanent failure merely
+because it occurs during startup.
+
+Keep hard dependencies on resources required to start the daemon, including
+its activation sockets. Failures in these prerequisites are outside the
+automatic recovery guarantee. Operators must repair the prerequisite and
+explicitly start the affected service. If a prerequisite has hit its own
+start or trigger limit, reset its failed state before starting it again.
+
Why the delay is fixed
----------------------
@@ -177,36 +289,86 @@ is a useful default alert threshold.
``RestartMode=direct`` makes failed-state monitoring even less useful: systemd
goes directly from a process failure to a restart without marking the unit as
failed or invoking ``OnFailure=`` units. Existing units may keep this setting,
-but it is not part of the restart policy.
+and services with activation sockets must use it as described above.
-Services that stop after status 6 or 9 remain in the failed state and should
-be covered by normal failed-unit monitoring. Journal output is useful for
-diagnosis but is not an alert.
+Services that stop after status 6 or 9 remain in the failed state and must
+be covered by failed-unit monitoring. Also monitor services that are expected
+to be running but are inactive, and failed prerequisite and socket units:
+a failed dependency start job need not mark the dependent service as failed.
+Expected state must account for intentional operator stops and maintenance.
+
+The service's ``NRestarts`` property counts automatic restarts, not independent
+activations. Socket-triggered starts can therefore form a tight loop while
+``NRestarts`` stays zero. Restart-frequency monitoring does not replace the
+activation controls above. Journal output is useful for diagnosis but is not
+an alert.
The relevant behavior is documented in `systemd.service(5)`_,
-`systemd.unit(5)`_, and the systemd 258 `RESTART_RESET notification`_.
+`systemd.unit(5)`_, `systemd.socket(5)`_, and the systemd 258
+`RESTART_RESET notification`_.
.. _systemd.service(5): https://manpages.debian.org/trixie/systemd/systemd.service.5.en.html
.. _systemd.unit(5): https://manpages.debian.org/trixie/systemd/systemd.unit.5.en.html
+.. _systemd.socket(5): https://manpages.debian.org/trixie/systemd/systemd.socket.5.en.html
.. _RESTART_RESET notification: https://manpages.debian.org/testing/libsystemd-dev/sd_pid_notify_barrier.3.en.html
Test Plan
=========
-Run ``systemd-analyze verify`` against the changed units. With a representative
-service, check that status 1 keeps restarting at ten-second intervals while
-statuses 6 and 9 do not restart. Check that repeated restarts trigger the
-deployment's crash-loop alert.
+Run ``systemd-analyze verify`` against the changed service and socket units
+together, including their dependency graph. Run the following acceptance
+tests on systemd 257 in Debian Trixie with representative services and all
+their associated activation sockets:
+
+* Exit statuses 0 and 1, termination by signals, and intentional
+ ``RuntimeMaxSec=`` recycling trigger automatic restarts with a ten-second
+ delay after termination. Activation sockets stay available during that
+ delay, and incoming traffic does not cause an earlier restart.
+* Exit statuses 6 and 9 leave the service failed and stop every associated
+ socket. Test with connections already pending and with subsequent traffic;
+ neither may cause further starts. Numeric exit statuses 6 and 9 must not
+ suppress restarts after ``SIGABRT`` or ``SIGKILL``.
+* Explicitly stopping the service stops its sockets and prevents further
+ starts under traffic, including when stopped during the restart delay.
+ Explicitly starting it again restores the service and its sockets.
+* Starting an associated socket starts the daemon without waiting for traffic.
+* Starting a product target with a failed reconnectable dependency does not
+ prevent the consumer from starting. When the dependency recovers, internal
+ reconnection or the next automatic restart restores useful service without
+ an operator restarting the consumer. If the dependency exits with status
+ 6 or 9 or is explicitly stopped, repeated consumer restarts must not start
+ it again.
+* Failed prerequisites, including a socket bind failure and a dependency
+ start-job failure, trigger an alert even if the service remains inactive.
+ Repairing the prerequisite and explicitly starting the service restores it.
+* Repeated automatic restarts trigger the deployment's crash-loop alert;
+ permanent failures trigger its failed-unit alert.
+
+Preliminary tests of the socket-lifetime profile on systemd 259 confirmed that
+statuses 6 and 9 stop the socket, transient failures preserve it without
+shortening the restart delay under traffic, and an explicit stop stops both
+units. Shared-target tests also confirmed that consumers recover when a
+dependency becomes available, while consumer restarts leave a dependency that
+exited with status 6 stopped. These checks do not replace the systemd 257
+acceptance tests above.
Definition of Done
==================
* [ ] All checked-in long-running product units follow the baseline policy.
+* [ ] Associated activation sockets follow the lifetime profile, including
+ permanent failure, explicit stop, and template-instance behavior.
* [ ] Distribution and deployment copies match their product units.
* [ ] Service implementations return status 6 for invalid or missing
configuration and reserve status 9 for known permanent failures.
+* [ ] Reconnectable dependencies do not block daemon startup or propagate
+ stops; their temporary unavailability remains restartable, and consumer
+ restarts do not reactivate stopped dependencies.
* [ ] Worker wrappers do not hide service termination from systemd.
-* [ ] Supported deployments detect stopped services and repeated restarts.
+* [ ] Supported deployments detect unexpectedly inactive services, failed
+ services and prerequisites, and repeated restarts, and document operator
+ recovery from prerequisite failures.
+* [ ] The service and socket acceptance tests pass on systemd 257.
* [ ] The accepted policy is added to the appropriate developer or operations
reference manual.
@@ -249,6 +411,12 @@ Unlimited retries can consume resources and produce repeated log messages,
which is why restart monitoring is required. Incorrectly returning status 6
or 9 can also turn a recoverable failure into one that waits for an operator.
+Binding activation sockets to the service starts the daemon as soon as a
+socket starts, losing startup deferred until incoming traffic. A permanent
+failure or explicit stop also closes those sockets, so clients cannot queue
+connections until an operator starts the service again. Failures in systemd
+prerequisites still require monitoring and operator recovery.
+
Discussion / Q&A
================
