Concrete systemd unit patterns that reduced flakiness: restart policies, resource limits, and structured logs.

Systemd Tricks We Use to Keep Services Boring

After a few painful outages caused by homemade init scripts, we moved everything to systemd and wrote down the patterns that worked.

Pattern: Restart with Backoff#

We had a service that occasionally failed to bind its port on boot.

```ini [Unit] Description=API service After=network-online.target Wants=network-online.target

[Service] ExecStart=/usr/local/bin/api Restart=on-failure RestartSec=5

[Install] WantedBy=multi-user.target ```

• Restart=on-failure + RestartSec gave the process room to recover without flapping.

Pattern: Non-Root with Limits#

We saw file descriptor exhaustion during load tests.

• Added User=api and LimitNOFILE=65536.
• Used Ansible to roll the unit file change across the fleet.

Pattern: Journald as a Timeline#

When something goes wrong, we start with:

• `journalctl -u api -b`
• `journalctl -u api --since "-15min"`

Systemd didn’t fix our code, but it made failures predictable and repeatable.