Debugging a systemd Service That Won't Start

A service that won't come up is one of those problems that looks scary and usually isn't. systemd already knows what went wrong. The whole job is getting it to tell you, then matching what it says to one of a handful of common causes. Here's the workflow I use, top to bottom, when a unit is dead on arrival.

Start with status#

The first command, every time:

systemctl status myapp.service

You care about two things in that output: the Active: line and the Process: / Main PID: lines near the bottom, plus the last few log entries systemd tacks on for free.

× myapp.service - My Application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2026-07-21 14:03:11 UTC; 8s ago
    Process: 4821 ExecStart=/opt/myapp/bin/myapp --config /etc/myapp.conf (code=exited, status=203/EXEC)
   Main PID: 4821 (code=exited, status=203/EXEC)
        CPU: 4ms

Jul 21 14:03:11 host systemd[1]: myapp.service: Main process exited, code=exited, status=203/EXEC
Jul 21 14:03:11 host systemd[1]: myapp.service: Failed with result 'exit-code'.

Read the Result: value. It tells you the category of failure before you read a single application log:

• exit-code: the process ran and returned non-zero. Could be your app, could be systemd failing to exec it.
• signal: the kernel or something else killed it (SIGSEGV, SIGABRT).
• timeout: it didn't signal readiness in time. Almost always a Type= mismatch.
• oom-kill: the OOM killer took it. Check MemoryMax= and actual usage.

The status= number matters too. Anything in the 2xx range is systemd's own exec machinery, not your program. 203/EXEC means the binary couldn't be executed (wrong path, not executable, wrong interpreter). 200/CHDIR means WorkingDirectory= is bad. 217/USER means the User= doesn't exist. Those codes point straight at the unit file, not the app.

Read the journal#

Status shows the tail. For the full story, go to the journal:

journalctl -u myapp.service -b --no-pager
journalctl -u myapp.service -e

-b scopes to the current boot, --no-pager dumps it straight to stdout for grepping, and -e jumps to the end interactively. When you're chasing a restart loop, -f follows live while you poke at it. If the app logs to stdout/stderr, systemd captures all of it here, so this is usually where the real error message lives.

The usual suspects#

Once you know the Result and status code, it's pattern matching. These seven cover the vast majority of what I've seen.

1. Bad ExecStart. status=203/EXEC almost always means the path in ExecStart= is wrong, not absolute, or not executable. systemd does not use $PATH, so ExecStart=myapp fails. Give it the full path and confirm the bit is set:

ls -l /opt/myapp/bin/myapp
file /opt/myapp/bin/myapp     # is it actually an ELF binary / right interpreter?
chmod +x /opt/myapp/bin/myapp

A missing shebang or a script pointing at /usr/bin/python that isn't there shows up as 203 too.

2. Working directory or user/group. 200/CHDIR means WorkingDirectory= points somewhere that doesn't exist or the service user can't reach. 217/USER means User= or Group= is undefined. Verify the account exists and the directory is readable by it. A service running as www-data can't cd into a root-only /root/app.

3. Missing environment. If the app needs config through env vars and you loaded them via EnvironmentFile=, a missing or unreadable file will either fail the unit or leave the app starved of what it needs. Note that EnvironmentFile=-/etc/default/myapp with the leading dash makes the file optional; without the dash a missing file is fatal. Confirm the file exists and the values look right.

4. Type mismatch. This is the sneaky one. Your app forks and daemonizes itself, but the unit says Type=simple. systemd expects the main process to stay in the foreground, so when the parent exits it thinks the service died. You get a timeout or an immediate failure. If the app backgrounds itself, use Type=forking. If it can signal readiness over sd_notify, use Type=notify, which is the most robust. Most modern apps run in foreground and want plain Type=simple.

5. Dependencies and ordering. The service needs the network up, or a database, or an NFS mount, and it started too early. After= controls ordering, Requires= (or the softer Wants=) controls whether the dependency is pulled in at all. A service that binds to a specific IP wants After=network-online.target plus Wants=network-online.target. One that reads from a mount wants RequiresMountsFor=.

6. Permissions and SELinux. The binary runs, the user is right, and it still dies opening a socket or a file. On SELinux systems check ausearch -m avc -ts recent or journalctl for AVC denied messages. A binary in a non-standard path often needs the right file context. On plain DAC systems it's usually a file mode or a privileged port under 1024 without AmbientCapabilities=CAP_NET_BIND_SERVICE.

7. Syntax and stale state. Verify the unit before you trust it, and reload after every edit:

systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl restart myapp.service

daemon-reload is the one people forget. Edit the file, restart, wonder why nothing changed, because systemd is still running the old in-memory copy.

See what systemd actually loaded#

Never trust the file on disk over what's running. systemctl cat shows the effective unit including any drop-ins:

systemctl cat myapp.service

For a targeted fix that survives package upgrades, use a drop-in override instead of editing the vendor file:

sudo systemctl edit myapp.service

That drops you into a override.conf under /etc/systemd/system/myapp.service.d/. A minimal example that fixes a couple of the causes above:

[Service]
Type=simple
ExecStart=
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp.conf
WorkingDirectory=/opt/myapp
User=myapp
EnvironmentFile=-/etc/default/myapp

The empty ExecStart= first clears the inherited value; without it, systemd rejects a second ExecStart on Type=simple.

Restart loops#

If the service flaps and then goes fully dead with start request repeated too quickly, it hit the StartLimit. systemd stops retrying after StartLimitBurst failures inside StartLimitIntervalSec. That message is not the root cause, it's the rate limiter. Fix the underlying failure, then clear the counter:

sudo systemctl reset-failed myapp.service
sudo systemctl start myapp.service

The call we'd make#

Work it in order and you rarely guess. Read systemctl status for the Result and the exit code first, because a 2xx status means the unit file is wrong and you can skip the app logs entirely. Pull the journal for anything else. Match the symptom to one of the seven causes, fix it with a drop-in rather than hacking the vendor file, and always daemon-reload before you restart. Nine times out of ten it's a non-absolute ExecStart, a Type= mismatch, or a missing dependency, and systemd told you which one on the very first command. For the wider playbook, see our Linux troubleshooting guide.