102-systemd-service-restart-policy.rst (20713B)
1 DD 102: systemd service restart policy 2 ###################################### 3 4 :Design status: Draft 5 :Implementation status: Not started 6 :DD shepherd: Florian Dold 7 :Historical contributors: Florian Dold 8 :First published: 2026-09-03 9 :Last substantive change: 2026-09-05 10 11 Summary 12 ======== 13 14 Long-running Taler services should recover from transient daemon failures 15 without operator intervention. They should stop on invalid configuration or 16 another known permanent failure, and repeated crashes must be visible to 17 operators. 18 19 The proposed policy retries indefinitely with a fixed ten-second delay. Exit 20 status 6 means that the service is not configured correctly; exit status 9 21 means that the service encountered another permanent failure. Neither status 22 causes an automatic restart. Associated activation sockets stop with the 23 service so that incoming traffic cannot start it again after a permanent 24 failure or an explicit stop. 25 26 This recovery guarantee assumes that systemd can start the daemon. Failures 27 in systemd prerequisites, such as a socket that cannot bind, require monitoring 28 and operator repair. 29 30 Systemd 257 in Debian Trixie supports an increasing restart delay, but does 31 not reset that delay after the service has been healthy for a long time. If a 32 service reaches a five-minute delay, runs successfully for days, and then 33 crashes again, the next restart still waits five minutes. Services restarted 34 regularly through ``RuntimeMaxSec=`` also advance the counter. A fixed delay 35 avoids this persistent state. 36 37 Motivation 38 ========== 39 40 Taler's systemd units currently use different restart policies. Some permit 41 only five starts in five seconds and some rely on systemd's default start-rate 42 limit. Once that limit is reached, ``Restart=always`` no longer restarts the 43 service. A temporary dependency outage can therefore require manual 44 recovery. 45 46 Other units restart after only a few milliseconds or seconds, producing a 47 tight loop during a longer outage. Some configure ``RestartSteps=`` without 48 the required maximum delay, so the setting has no effect. 49 50 Exit handling is inconsistent as well. Native Taler services commonly use 51 status 6 for configuration errors and status 9 for failures that should not be 52 retried, but not all units recognize both statuses and non-C services do not 53 always return them. 54 55 Socket activation and unit dependencies also affect recovery. Incoming 56 traffic can start a service again after ``RestartPreventExitStatus=`` has 57 suppressed its automatic restart, without observing ``RestartSec=``. In the 58 other direction, a failed dependency with ``Requires=`` and ``After=`` can 59 prevent the daemon from starting at all, leaving its restart policy unused. 60 61 An unlimited restart policy also needs monitoring. In particular, 62 ``RestartMode=direct`` skips the failed/inactive transition during automatic 63 restarts and does not invoke ``OnFailure=`` units. A service can therefore 64 remain in a crash loop without triggering monitoring based only on its current 65 state. 66 67 Requirements 68 ============ 69 70 * Long-running services keep retrying after transient daemon failures, 71 including temporary loss of reconnectable dependencies. 72 * Restarts do not form a tight loop. 73 * Invalid configuration and known permanent failures are not retried. 74 * Exit statuses have the same meaning in every implementation language. 75 * An explicit ``systemctl stop`` stops the service and its activation sockets. 76 * Operators can detect stopped services, failed prerequisites, and crash loops. 77 * The policy works with Debian Trixie. 78 79 Proposed Solution 80 ================= 81 82 Scope 83 ----- 84 85 The policy applies to long-running product services shipped or deployed by 86 Taler, Anastasis, LibEuFin, Donau, Paivana, Challenger, and related 87 repositories, including distribution-specific copies of their units. 88 The socket-lifetime rules below also apply to their associated activation 89 sockets. 90 91 The automatic recovery guarantee covers failures handled by the daemon and 92 its service restart policy. It does not cover failures in systemd 93 prerequisites, such as failed assertions, socket bind failures, or dependency 94 start-job failures. These may leave a service inactive without ever running 95 its main process. Deployments must detect these cases and provide operator 96 recovery; disabling the service start-rate limiter does not resolve them. 97 98 The service baseline does not apply to one-shot initialization commands, 99 timer-triggered jobs, garbage collection jobs, or other processes that are 100 expected to finish. 101 102 Unit policy 103 ----------- 104 105 Long-running services use this baseline: 106 107 .. code-block:: ini 108 109 [Unit] 110 StartLimitIntervalSec=0 111 112 [Service] 113 Restart=always 114 RestartSec=10s 115 RestartPreventExitStatus=6 9 116 117 ``StartLimitIntervalSec=0`` disables the service's start-rate limiter. This 118 is needed because ``Restart=`` remains subject to start-rate limiting. 119 120 ``Restart=always`` restarts the service after clean and unsuccessful exits, 121 signals, timeouts, and watchdog failures. It does not override an explicit 122 ``systemctl stop``. 123 124 ``RestartSec=10s`` avoids a busy loop and has no backoff state to carry across 125 unrelated failures. Units must not add ``StartLimitBurst``, a non-zero 126 ``StartLimitIntervalSec``, the legacy ``StartLimitInterval``, 127 ``RestartSteps``, or ``RestartMaxDelaySec``. A unit may prevent additional 128 exit statuses when its program documents them as permanent failures. 129 130 Socket lifetime 131 --------------- 132 133 For a service with activation sockets, apply the baseline above and add the 134 following to ``example.service``: 135 136 .. code-block:: ini 137 138 [Unit] 139 Requires=example.socket 140 141 [Service] 142 RestartMode=direct 143 144 Its ``example.socket`` uses this lifetime relationship, in addition to its 145 existing listen address and access settings: 146 147 .. code-block:: ini 148 149 [Unit] 150 BindsTo=example.service 151 152 [Socket] 153 Accept=no 154 Service=example.service 155 156 Apply this relationship to every activation socket of the service, including 157 the corresponding instances of template units. The service requires all 158 sockets whose file descriptors it needs. Do not add ``After=example.service`` 159 to a socket: systemd already orders activation sockets before their services, 160 so the opposite ordering would introduce a cycle. 161 162 ``BindsTo=`` stops the sockets when the service stops or fails permanently. 163 ``RestartMode=direct`` is required for this profile: it avoids the transient 164 failed/inactive state during automatic restarts, so those restarts keep the 165 sockets available. Pending connections wait for the next restart; they must 166 not shorten the ten-second delay. ``PartOf=`` alone does not provide the 167 required behavior on permanent failure. 168 169 Starting a socket with this relationship also starts the daemon. These are 170 long-running services, rather than daemons started only when traffic arrives. 171 After exit status 6 or 9, the service remains failed and its sockets stop. 172 An explicit ``systemctl stop example.service`` also stops the sockets, so 173 subsequent traffic cannot reactivate the service. After repairing the failure, 174 ``systemctl start example.service`` starts the service and its required sockets. 175 176 ``RestartPreventExitStatus=`` only suppresses automatic service restarts. It 177 does not reject independent start requests, including socket activation or an 178 operator's explicit start. Deployments must not add unconditional activation 179 mechanisms, such as ``Upholds=`` or a handler that always starts the service, 180 where they would override permanent exits or explicit stops. Socket trigger 181 limits are separate from the service start-rate limiter and are not a 182 substitute for the lifetime relationship above. 183 184 Reconnectable dependencies 185 -------------------------- 186 187 For dependencies that the daemon can reconnect to, such as a database or 188 security module, request local startup through a shared product target and 189 order startup with ``After=`` on the consumer. For example, 190 ``example.target`` uses ``Wants=example.service example-dependency.service``, 191 while ``example.service`` uses ``After=example-dependency.service``. A failed 192 dependency then does not prevent the consumer from starting. 193 194 Replace ``Requires=`` or ``BindsTo=`` relationships from the consumer that 195 would block its startup or stop it when a reconnectable dependency goes away. 196 A failed start job with ``Requires=`` and ``After=`` does not schedule an 197 automatic restart of the consumer; starting the dependency later does not 198 automatically start that service either. 199 200 Do not simply replace these relationships with ``Wants=`` on the consumer. 201 An automatic restart of the consumer also pulls in its wanted units, which 202 can start a dependency again after a permanent exit or explicit stop. With 203 ``Wants=`` on the shared target instead, consumer restarts do not reactivate 204 the dependency. Start the product target to request startup of the group; 205 starting the consumer alone does not start its reconnectable dependencies. 206 207 The daemon must handle an unavailable dependency by reconnecting internally 208 or returning a restartable status, such as 1. Temporary unavailability must 209 not be classified as invalid configuration or a permanent failure merely 210 because it occurs during startup. 211 212 Keep hard dependencies on resources required to start the daemon, including 213 its activation sockets. Failures in these prerequisites are outside the 214 automatic recovery guarantee. Operators must repair the prerequisite and 215 explicitly start the affected service. If a prerequisite has hit its own 216 start or trigger limit, reset its failed state before starting it again. 217 218 Why the delay is fixed 219 ---------------------- 220 221 Systemd 257 calculates its increasing delay from the number of automatic 222 restarts. It has no setting equivalent to "reset the delay after ten minutes 223 of successful operation". Time spent active does not reduce the counter. 224 The counter is reset only by an explicit manager action such as 225 ``systemctl reset-failed`` or by stopping and starting the unit. 226 227 This matters even for services that rarely fail. For example, after enough 228 failures to reach a five-minute cap: 229 230 #. the service recovers and runs for a week; 231 #. it encounters one unrelated transient failure; 232 #. systemd still waits five minutes before restarting it. 233 234 It is worse for a service using ``RuntimeMaxSec=`` as an intentional recycling 235 mechanism: every scheduled automatic restart advances the same counter, so the 236 normal restart delay eventually reaches the cap. 237 238 Systemd 258 adds ``RESTART_RESET=1`` to its service notification protocol. A 239 daemon can send this after it considers itself healthy, but systemd 257 ignores 240 it and every daemon would need explicit support. The policy can be revisited 241 once the minimum supported systemd version and the applications can provide a 242 reliable reset. 243 244 Exit-status contract 245 -------------------- 246 247 The main process uses these common statuses: 248 249 ``6`` 250 Configuration is missing or invalid. Retrying the same configuration 251 cannot succeed. 252 253 ``9`` 254 The service has identified another permanent failure and asks not to be 255 restarted. 256 257 All other failures are restartable. Implementations in C, Kotlin, Rust, Go, 258 and other languages must map configuration failures to status 6 rather than a 259 generic status 1. Before returning 6 or 9, the program should log a clear 260 diagnostic. 261 262 ``RestartPreventExitStatus`` applies only to the main service process. It 263 does not affect ``ExecStartPre=``. Configuration validation that decides 264 whether the service should be retried must therefore be reflected in the main 265 process's exit status. 266 267 Application-level retry 268 ----------------------- 269 270 Restarting the process is the fallback when it cannot continue. If a daemon 271 can keep serving useful work or report degraded health while reconnecting to a 272 dependency, it should remain running and retry that operation itself. 273 Domain-specific waits need not use the systemd restart interval. 274 275 A service should not wrap its main worker in an internal relaunch loop. Such 276 a loop hides the worker's exit status and restart count from systemd. If the 277 worker's termination makes the service unavailable, propagate the result and 278 let systemd restart it. 279 280 A failed synchronization pass is not necessarily worker termination. Keep 281 recoverable API and database-operation retries inside the worker, with suitable 282 backoff that resets after success. Reconnect failed sessions without discarding 283 healthy clients or independent work. For example, Wise balances have independent 284 polling/retry loops; Cyclos polling continues while its optional notification 285 listener reconnects. The adapters' HTTP servers are separate services, so a 286 worker restart does not restart HTTP serving. Propagate terminal failures such 287 as an unusable closed database pool, and do not catch panics merely to relaunch 288 the application. 289 290 Detecting crash loops 291 --------------------- 292 293 A service that crashes repeatedly may still appear active whenever monitoring 294 checks it, because systemd keeps restarting it. Checking only whether the 295 unit is active or failed therefore misses crash loops. Deployments must also 296 monitor how often a service restarts. Three restarts within fifteen minutes 297 is a useful default alert threshold. 298 299 ``RestartMode=direct`` makes failed-state monitoring even less useful: systemd 300 goes directly from a process failure to a restart without marking the unit as 301 failed or invoking ``OnFailure=`` units. Existing units may keep this setting, 302 and services with activation sockets must use it as described above. 303 304 Services that stop after status 6 or 9 remain in the failed state and must 305 be covered by failed-unit monitoring. Also monitor services that are expected 306 to be running but are inactive, and failed prerequisite and socket units: 307 a failed dependency start job need not mark the dependent service as failed. 308 Expected state must account for intentional operator stops and maintenance. 309 310 The service's ``NRestarts`` property counts automatic restarts, not independent 311 activations. Socket-triggered starts can therefore form a tight loop while 312 ``NRestarts`` stays zero. Restart-frequency monitoring does not replace the 313 activation controls above. Journal output is useful for diagnosis but is not 314 an alert. 315 316 The relevant behavior is documented in `systemd.service(5)`_, 317 `systemd.unit(5)`_, `systemd.socket(5)`_, and the systemd 258 318 `RESTART_RESET notification`_. 319 320 .. _systemd.service(5): https://manpages.debian.org/trixie/systemd/systemd.service.5.en.html 321 .. _systemd.unit(5): https://manpages.debian.org/trixie/systemd/systemd.unit.5.en.html 322 .. _systemd.socket(5): https://manpages.debian.org/trixie/systemd/systemd.socket.5.en.html 323 .. _RESTART_RESET notification: https://manpages.debian.org/testing/libsystemd-dev/sd_pid_notify_barrier.3.en.html 324 325 Test Plan 326 ========= 327 328 Run ``systemd-analyze verify`` against the changed service and socket units 329 together, including their dependency graph. Run the following acceptance 330 tests on systemd 257 in Debian Trixie with representative services and all 331 their associated activation sockets: 332 333 * Exit statuses 0 and 1, termination by signals, and intentional 334 ``RuntimeMaxSec=`` recycling trigger automatic restarts with a ten-second 335 delay after termination. Activation sockets stay available during that 336 delay, and incoming traffic does not cause an earlier restart. 337 * Exit statuses 6 and 9 leave the service failed and stop every associated 338 socket. Test with connections already pending and with subsequent traffic; 339 neither may cause further starts. Numeric exit statuses 6 and 9 must not 340 suppress restarts after ``SIGABRT`` or ``SIGKILL``. 341 * Explicitly stopping the service stops its sockets and prevents further 342 starts under traffic, including when stopped during the restart delay. 343 Explicitly starting it again restores the service and its sockets. 344 * Starting an associated socket starts the daemon without waiting for traffic. 345 * Starting a product target with a failed reconnectable dependency does not 346 prevent the consumer from starting. When the dependency recovers, internal 347 reconnection or the next automatic restart restores useful service without 348 an operator restarting the consumer. If the dependency exits with status 349 6 or 9 or is explicitly stopped, repeated consumer restarts must not start 350 it again. 351 * Failed prerequisites, including a socket bind failure and a dependency 352 start-job failure, trigger an alert even if the service remains inactive. 353 Repairing the prerequisite and explicitly starting the service restores it. 354 * Repeated automatic restarts trigger the deployment's crash-loop alert; 355 permanent failures trigger its failed-unit alert. 356 357 Preliminary tests of the socket-lifetime profile on systemd 259 confirmed that 358 statuses 6 and 9 stop the socket, transient failures preserve it without 359 shortening the restart delay under traffic, and an explicit stop stops both 360 units. Shared-target tests also confirmed that consumers recover when a 361 dependency becomes available, while consumer restarts leave a dependency that 362 exited with status 6 stopped. These checks do not replace the systemd 257 363 acceptance tests above. 364 365 Definition of Done 366 ================== 367 368 * [ ] All checked-in long-running product units follow the baseline policy. 369 * [ ] Associated activation sockets follow the lifetime profile, including 370 permanent failure, explicit stop, and template-instance behavior. 371 * [ ] Distribution and deployment copies match their product units. 372 * [ ] Service implementations return status 6 for invalid or missing 373 configuration and reserve status 9 for known permanent failures. 374 * [ ] Reconnectable dependencies do not block daemon startup or propagate 375 stops; their temporary unavailability remains restartable, and consumer 376 restarts do not reactivate stopped dependencies. 377 * [ ] Worker wrappers do not hide service termination from systemd. 378 * [ ] Supported deployments detect unexpectedly inactive services, failed 379 services and prerequisites, and repeated restarts, and document operator 380 recovery from prerequisite failures. 381 * [ ] The service and socket acceptance tests pass on systemd 257. 382 * [ ] The accepted policy is added to the appropriate developer or operations 383 reference manual. 384 385 Alternatives 386 ============ 387 388 Finite start-rate limit 389 ----------------------- 390 391 Systemd's default limit protects the host and leaves a persistently failing 392 unit in a clear failed state. It also abandons recovery after a temporary 393 outage, so it does not meet the main requirement. 394 395 Systemd exponential delay 396 ------------------------- 397 398 An increasing delay with a cap reduces load during a long outage. On systemd 399 257, however, the delay does not reset after a healthy period. This makes an 400 old failure history determine recovery time for a later, unrelated failure. 401 It should be reconsidered when a reliable healthy-runtime reset is available. 402 403 Timer, wrapper, or external orchestrator 404 ---------------------------------------- 405 406 A timer or wrapper can implement a custom reset rule, but duplicates systemd 407 supervision and complicates stop, status, and exit-code handling. An external 408 orchestrator can provide richer restart policies, but requiring one would 409 change Taler's deployment model. Deployments that already use an orchestrator 410 may translate the behavior specified here into its native policy. 411 412 Drawbacks 413 ========= 414 415 A ten-second delay is not adaptive. A single crash recovers more slowly than 416 with the shortest current settings, while a long outage causes more attempts 417 than capped exponential backoff. It also adds ten seconds to an intentional 418 ``RuntimeMaxSec`` restart. 419 420 Unlimited retries can consume resources and produce repeated log messages, 421 which is why restart monitoring is required. Incorrectly returning status 6 422 or 9 can also turn a recoverable failure into one that waits for an operator. 423 424 Binding activation sockets to the service starts the daemon as soon as a 425 socket starts, losing startup deferred until incoming traffic. A permanent 426 failure or explicit stop also closes those sockets, so clients cannot queue 427 connections until an operator starts the service again. Failures in systemd 428 prerequisites still require monitoring and operator recovery. 429 430 Discussion / Q&A 431 ================ 432 433 Why restart after a clean exit? 434 ------------------------------- 435 436 These units represent services expected to remain available. Processes that 437 are expected to finish belong in one-shot or timer-driven units. 438 439 Why not treat every startup failure as permanent? 440 ------------------------------------------------- 441 442 Startup may depend on a database, network service, or credential agent that 443 is temporarily unavailable. Only failures classified by the program as 444 invalid configuration or permanent should suppress restart.