Debugging a systemd Service That Won't Start

A failed systemd unit almost always tells you why on the first command: systemctl status myapp shows the Result= category and the exit status, and journalctl -u myapp -b shows the full error. A status in the 2xx range (203/EXEC, 217/USER, 200/CHDIR) means the unit file is broken, not your app.

The workflow we use, top to bottom, when a unit is dead on arrival:

1. systemctl --failed to list every unit in a failed state.
2. systemctl status myapp.service for the Result: and exit status.
3. journalctl -u myapp.service -n 50 --no-pager for the real error.
4. Run the ExecStart= command by hand, as the service user, to reproduce it.

Start with status

$ systemctl status myapp.service
× myapp.service - My Application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2026-07-21 14:03:11 UTC; 8s ago
    Process: 4821 ExecStart=/opt/myapp/bin/myapp --config /etc/myapp.conf (code=exited, status=203/EXEC)
   Main PID: 4821 (code=exited, status=203/EXEC)

Jul 21 14:03:11 host systemd[1]: myapp.service: Main process exited, code=exited, status=203/EXEC
Jul 21 14:03:11 host systemd[1]: myapp.service: Failed with result 'exit-code'.

Read the Result: value. It tells you the category of failure before you read a single application log:

• exit-code: the process ran and returned non-zero. Could be your app, could be systemd failing to exec it.
• signal: the kernel or something else killed it (SIGSEGV, SIGABRT).
• timeout: it didn't signal readiness within TimeoutStartSec=, 90 seconds by default. Almost always a Type= mismatch.
• oom-kill: the OOM killer took it. Check MemoryMax= and actual usage.

The status= number matters too. Anything in the 2xx range comes from systemd's own exec machinery, before your program ran:

Read the journal

Status shows the tail. For the full story:

journalctl -u myapp.service -b --no-pager
journalctl -u myapp.service -f

-b scopes to the current boot, --no-pager dumps straight to stdout for grepping, and -f follows live while you poke at a restart loop. If the app logs to stdout/stderr, systemd captures all of it here, so this is usually where the real error message lives.

The usual suspects

Once you know the Result and status code, it's pattern matching. These seven cover the vast majority of what we've seen.

1. Bad ExecStart. 203/EXEC almost always means the path in ExecStart= is wrong or not executable. systemd resolves a bare name against a fixed compile-time search path, not your shell's $PATH, so give it the full path and confirm the bit is set:

ls -l /opt/myapp/bin/myapp
file /opt/myapp/bin/myapp     # ELF binary, or a script with a valid shebang?
chmod +x /opt/myapp/bin/myapp

A missing shebang or a script pointing at a /usr/bin/python that isn't there shows up as 203 too.

2. Working directory or user. 200/CHDIR and 217/USER, per the table. A service running as www-data can't cd into a root-only /root/app.

3. Missing environment. A missing EnvironmentFile= is fatal unless prefixed with a dash: EnvironmentFile=-/etc/default/myapp makes it optional. Confirm the file exists and the values look right.

4. Type mismatch. Your app forks and daemonizes itself, but the unit says Type=simple. systemd expects the main process to stay in the foreground, so when the parent exits it thinks the service died. If the app backgrounds itself, use Type=forking. If it supports sd_notify, use Type=notify, which gives systemd an accurate readiness signal. Most modern apps run in the foreground and want plain Type=simple.

5. Dependencies and ordering. The service started before the network, a database, or an NFS mount. After= controls ordering, Requires= or Wants= controls whether the dependency is pulled in. A service binding a specific IP wants After=network-online.target plus Wants=network-online.target. One that reads from a mount wants RequiresMountsFor=.

6. Permissions and SELinux. The binary runs, the user is right, and it still dies opening a file or socket. On SELinux systems check ausearch -m avc -ts recent for AVC denied. On plain DAC systems it's usually a file mode or a port under 1024 without AmbientCapabilities=CAP_NET_BIND_SERVICE.

7. Syntax and stale state. Verify the unit, and reload after every edit:

systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl restart myapp.service

daemon-reload is the one people forget: systemd keeps running the old in-memory copy until you reload.

See what systemd actually loaded

systemctl cat myapp.service shows the effective unit including drop-ins. For a fix that survives package upgrades, use sudo systemctl edit myapp.service, which writes override.conf under /etc/systemd/system/myapp.service.d/:

# /etc/systemd/system/myapp.service.d/override.conf
[Service]
Type=simple
ExecStart=
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp.conf
WorkingDirectory=/opt/myapp
User=myapp
EnvironmentFile=-/etc/default/myapp

The empty ExecStart= first clears the inherited value; without it, systemd rejects a second ExecStart on a non-oneshot service.

Questions people ask

What does status=203/EXEC mean in systemctl?

It means systemd could not execute the program in ExecStart=, so your code never ran. The usual causes are a relative or wrong path, a file without the execute bit, a script with no shebang or a shebang pointing at a missing interpreter, or SELinux blocking a binary in an unusual directory. Check with ls -l and file on the exact path in the unit.

How do I fix "start request repeated too quickly"?

That message is the rate limiter, not the root cause. By default systemd allows 5 starts within 10 seconds (DefaultStartLimitBurst=5, DefaultStartLimitIntervalSec=10s); a unit with Restart= that keeps crashing hits it and stops retrying. Find the real failure in journalctl -u NAME, fix it, then run systemctl reset-failed NAME and start the unit again.

Why does my service work by hand but fail under systemd?

systemd starts it with a different environment: no login shell profile, a minimal PATH, the User= from the unit instead of you, and a working directory of / unless set. Reproduce it faithfully with sudo -u myapp env -i /opt/myapp/bin/myapp --config /etc/myapp.conf, and move any variables you rely on into Environment= or EnvironmentFile=.

The call we'd make

Read systemctl status for the Result and exit code first, because a 2xx status means the unit file is wrong and you can skip the app logs entirely. Pull the journal for anything else, fix it with a drop-in rather than editing the vendor file, and always daemon-reload before you restart. Nine times out of ten it's a bad ExecStart, a Type= mismatch, or a missing dependency. For the wider playbook, see our Linux troubleshooting guide.

Sources

• systemd.exec(5): process exit codes and execution environment
• systemd.unit(5): StartLimitIntervalSec, StartLimitBurst, RequiresMountsFor
• systemd-system.conf(5): default start timeout and start-limit values
• systemd.service(5): Type=, ExecStart= and Restart=