Skip to main content
Operational Checklist: Systemd Service Reliability Patterns

Operational Checklist: Systemd Service Reliability Patterns

KU
· Founder & Chief Editor
• 7 min read•Updated •

A condensed checklist of the systemd unit-file patterns we now use everywhere, with the production reasons each one matters.

Every Linux service we run is managed by systemd. After years of accumulating war stories, our standard unit file template has stabilized around a small set of patterns. This is the checklist version: each item is something we've added because of a specific production incident, with a brief explanation of what it prevents.

This isn't a tutorial on systemd basics. It assumes you know what [Unit], [Service], and [Install] are.

The template, annotated

INI
[Unit]
Description=My Service
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=myservice
Group=myservice
WorkingDirectory=/opt/myservice

EnvironmentFile=/etc/myservice.env
ExecStart=/opt/myservice/bin/myservice

# Restart behaviour
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=10min
StartLimitBurst=3

# Resource limits
LimitNOFILE=65536
LimitNPROC=4096
MemoryMax=2G
CPUQuota=200%
TasksMax=512

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myservice

# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myservice /var/log/myservice
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictSUIDSGID=true
LockPersonality=true

# Watchdog (if the service supports sd_notify)
WatchdogSec=30
NotifyAccess=main

# Failure hook
OnFailure=alert-pagerduty@%n.service

[Install]
WantedBy=multi-user.target

The remainder of this post is each block explained.

After=network-online.target and Wants= paired

code
After=network-online.target
Wants=network-online.target

Without this, services that need DNS or external connectivity often start before the network is ready, fail their first connection, and crash-loop briefly. We had a service that took 30 seconds to come up after every reboot because it failed its initial DNS lookup, restarted, and only then succeeded.

Both lines are needed. Wants= declares the dependency; After= orders the units. Just After= without Wants= doesn't actually pull network-online.target into the boot sequence.

Type=exec not Type=simple

code
Type=exec

Type=simple (the default) considers the service started the instant the process is forked. Type=exec waits until execve() returns. The difference matters when a service crashes immediately on startup — with simple, systemd marks it active for a few milliseconds before the crash. With exec, the failure is visible.

This matters for systemctl start myservice && echo started behaving correctly.

User= and Group= always

code
User=myservice
Group=myservice

Never run as root. The user is created at install time. If the service needs to bind to a privileged port, we use AmbientCapabilities=CAP_NET_BIND_SERVICE rather than running as root.

Catching root processes in systemd-cgls is one of the easier audits we do periodically. Anything we find gets fixed.

Restart, but bounded

code
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=10min
StartLimitBurst=3

This restarts the service on failure (after 5 seconds), but not infinitely. If the service crashes 3 times in 10 minutes, systemd gives up and leaves it failed.

Why bound it: a service that's misconfigured will crash-loop forever otherwise, generating logs faster than your aggregation can handle, hammering the DB on each startup attempt, and never alerting anyone because "it's running fine — for 4 seconds at a time."

Three retries in 10 minutes is enough to handle transient issues (network blip, brief resource pressure) and stops short on genuine config errors.

Resource limits

code
LimitNOFILE=65536
LimitNPROC=4096
MemoryMax=2G
CPUQuota=200%
TasksMax=512

LimitNOFILE: file descriptors. The default is often 1024, which is too low for any service that handles many connections. We had a service drop incoming requests after exactly 1015 concurrent connections; took half a day to diagnose.

MemoryMax: hard limit, OOM-killed if exceeded. Usually set to slightly above the service's expected working set. Prevents one runaway service from taking down the whole node.

CPUQuota=200% means up to 2 CPUs worth. Useful on multi-tenant nodes; not strictly necessary on single-purpose hosts.

TasksMax: hard cap on threads/processes. On a service with a thread leak, the cap turns an unbounded leak into a controlled crash and alert.

Logging to journal, with identifier

code
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myservice

Sends both stdout and stderr to journald with a tag. journalctl -u myservice pulls them all. journalctl -t myservice works too.

Before this was standard, we had services writing to /var/log/myservice/myservice.log directly. Rotation was a separate config; sometimes it broke; sometimes the disk filled. The journald path is uniform across services and rotates automatically.

Hardening: the namespace and protection settings

code
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myservice /var/log/myservice
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true

These are systemd's built-in process isolation features. Each one closes a class of attack:

  • NoNewPrivileges: a compromised process can't escalate via setuid binaries.
  • PrivateTmp: each service gets its own /tmp, so a temp-file attack between services is impossible.
  • ProtectSystem=strict: most of the filesystem is mounted read-only for the service. We then explicitly list paths it can write to via ReadWritePaths.
  • ProtectHome: /home is invisible to the service.
  • ProtectKernel*: the service can't fiddle with kernel tunables, load modules, or modify cgroups.

Adding ProtectSystem=strict forces you to enumerate every directory each service legitimately writes to, and that audit tends to surface surprises, such as services writing to /var/lib/dpkg (wrong) or /etc/cron.d (definitely wrong).

The watchdog (when applicable)

code
WatchdogSec=30
NotifyAccess=main

For services that integrate with sd_notify, the watchdog kills (and restarts via the Restart= rule) any service that hasn't reported alive within 30 seconds. Useful for catching a process that's hung in a deadlock — the kernel sees the process as alive, but it's not making progress.

Most of our Go services use https://pkg.go.dev/github.com/coreos/go-systemd/daemon to send watchdog pings every 10 seconds.

OnFailure for alerting

code
OnFailure=alert-pagerduty@%n.service

When the service enters failed state, this triggers a separate template unit alert-pagerduty@.service which sends a PagerDuty event. The %n is the name of the failed unit.

The alert template is:

INI
[Unit]
Description=Page on failure of %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/page-pagerduty.sh %i

This means we get paged immediately on any failed service, with the failed unit's name in the alert. No need for separate monitoring; the OS itself is the alerting source.

Common mistakes we still see

People hitting the team with these issues:

Service runs fine manually but fails under systemd. Almost always an environment variable issue. The shell has env vars systemd doesn't. Use EnvironmentFile= for everything; never rely on inherited env.

Service refuses to start after daemon-reload. The unit file syntax error wasn't reported. Run systemd-analyze verify /etc/systemd/system/myservice.service to surface syntax issues.

Restart=always causing crash loops to go unnoticed. Don't use Restart=always. Use Restart=on-failure and bound it with StartLimitBurst.

Permissions issues on hardened services. ProtectSystem=strict means most dirs are read-only. Check the service's actual write paths and explicitly allow them via ReadWritePaths.

What we do once per quarter

Every quarter, run a script that audits every unit file on the fleet against this template. The audit reports:

  • Services running as root (we should have zero)
  • Services without NoNewPrivileges=true
  • Services without resource limits
  • Services without OnFailure=

The drift is small but real — new services occasionally ship without the full template. The quarterly audit catches them before they cause an incident.

What we don't bother with

Type=notify for services that don't natively support it. Wrapping a non-sd_notify-aware binary just to use notify mode is more trouble than it's worth.

Per-service seccomp filters via SystemCallFilter=. Powerful, but the failure modes are subtle (a single missed syscall and the service silently dies). We use the defaults that come with ProtectKernelTunables=true etc, which include reasonable seccomp restrictions.

Sandboxing via RootDirectory= to chroot the service. Useful for genuinely untrusted code; overkill for our own services where the threat model is "compromised dependencies," not "the binary is hostile."

What this template doesn't replace

Health checks (the service still needs to expose /healthz and Kubernetes/etc still need to consume it).

Application logging (journald is good for systemd-level events; structured app logging usually still goes to a separate aggregator).

Deployment automation. systemctl restart myservice after replacing the binary is the simple case; for safer rollouts you still need a deployer that watches health.

The systemd template is the foundation. Everything above it (k8s, prometheus, deployment tooling) builds on assumed-good service-level reliability. Get this layer right and the rest gets easier.

Explore topics:LinuxMonitoring
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2025). Operational Checklist: Systemd Service Reliability Patterns. DevOpsNess. Updated 2026-11-20. https://www.devopsness.com/blog/operational-checklist-systemd-service-reliability-patterns

Markdown

[Operational Checklist: Systemd Service Reliability Patterns](https://www.devopsness.com/blog/operational-checklist-systemd-service-reliability-patterns) (Kiril Urbonas, DevOpsNess, updated 2026-11-20)

KU

Kiril Urbonas

Founder & Chief Editor

608 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.