# Systemd Tricks We Use to Keep Services Boring

URL: https://www.devopsness.com/blog/systemd-tricks-we-use-to-keep-services-boring
Author: Kiril Urbonas
Published: 2025-03-05
Updated: 2025-03-05
Categories: Linux
Tags: Terraform, Ansible, Linux

Summary: Concrete systemd unit patterns that reduced flakiness: restart policies, resource limits, and structured logs.

---
# Systemd Tricks We Use to Keep Services Boring

After a few painful outages caused by homemade init scripts, we moved everything to systemd and wrote down the patterns that worked.

## Pattern: Restart with Backoff

We had a service that occasionally failed to bind its port on boot.

\`\`\`ini
[Unit]
Description=API service
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/api
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
\`\`\`

- **Restart=on-failure** + **RestartSec** gave the process room to recover without flapping.

## Pattern: Non-Root with Limits

We saw file descriptor exhaustion during load tests.

- Added **User=api** and **LimitNOFILE=65536**.
- Used Ansible to roll the unit file change across the fleet.

## Pattern: Journald as a Timeline

When something goes wrong, we start with:

- \`journalctl -u api -b\`
- \`journalctl -u api --since "-15min"\`

Systemd didn’t fix our code, but it made failures predictable and repeatable.