Systemd Tricks We Use to Keep Services Boring
Concrete systemd unit patterns that reduced flakiness: restart policies, resource limits, and structured logs.
Key takeaways
Concrete systemd unit patterns that reduced flakiness: restart policies, resource limits, and structured logs.
On this page
Systemd Tricks We Use to Keep Services Boring#
After a few painful outages caused by homemade init scripts, we moved everything to systemd and wrote down the patterns that worked.
Pattern: Restart with Backoff#
We had a service that occasionally failed to bind its port on boot.
```ini [Unit] Description=API service After=network-online.target Wants=network-online.target
[Service] ExecStart=/usr/local/bin/api Restart=on-failure RestartSec=5
[Install] WantedBy=multi-user.target ```
- Restart=on-failure + RestartSec gave the process room to recover without flapping.
Pattern: Non-Root with Limits#
We saw file descriptor exhaustion during load tests.
- Added User=api and LimitNOFILE=65536.
- Used Ansible to roll the unit file change across the fleet.
Pattern: Journald as a Timeline#
When something goes wrong, we start with:
- `journalctl -u api -b`
- `journalctl -u api --since "-15min"`
Systemd didn’t fix our code, but it made failures predictable and repeatable.
Get the DevOps Troubleshooting Cheat Sheet
Subscribe and get our free one-page reference for the errors that eat an afternoon — CrashLoopBackOff, OOMKilled, Terraform state locks, and more — plus new guides as we publish them.
A Pragmatic Multi-Region Strategy for Small Teams
How a small team moved from single-region risk to a simple active/passive multi-region setup without doubling complexity.
How We Stopped Terraform Drift from Surprising On-Call
A real story of removing console-only changes, adding drift detection, and getting Terraform back in charge.
More from Linux
Explore more articles in this category
CERN Left Red Hat for Debian: The CPU Baseline Lesson
CERN is moving accelerator control computers to Debian because RHEL raised its minimum CPU level. Check your own fleet's x86-64-vN support before the next OS upgrade does it for you.
ext4 vs XFS vs Btrfs: Choosing a Filesystem for a Server
The default filesystem your distro picks is not always the right one for your workload. Here is what actually differs and when each one wins.
journald Log Management: Retention, Filtering, and Forwarding
journald is the default log sink on every systemd distro, and most of it runs on defaults nobody chose. Here is how to actually control it.
You might have missed
Evergreen posts worth revisiting.
