# Debugging a systemd Service That Won't Start

URL: https://www.devopsness.com/blog/systemd-service-fails-to-start
Author: Kiril Urbonas
Published: 2026-07-22
Updated: 2026-10-03
Categories: Linux
Tags: Linux, Systemd, Troubleshooting, Sysadmin, Services

Summary: A field-tested workflow for diagnosing why a systemd unit refuses to start, from status output to exit codes to the usual root causes.

---
A failed systemd unit almost always tells you why on the first command: `systemctl status myapp` shows the `Result=` category and the exit status, and `journalctl -u myapp -b` shows the full error. A status in the 2xx range (`203/EXEC`, `217/USER`, `200/CHDIR`) means the unit file is broken, not your app.

The workflow we use, top to bottom, when a unit is dead on arrival:

1. `systemctl --failed` to list every unit in a failed state.
2. `systemctl status myapp.service` for the `Result:` and exit status.
3. `journalctl -u myapp.service -n 50 --no-pager` for the real error.
4. Run the `ExecStart=` command by hand, as the service user, to reproduce it.

## Start with status

```
$ systemctl status myapp.service
× myapp.service - My Application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2026-07-21 14:03:11 UTC; 8s ago
    Process: 4821 ExecStart=/opt/myapp/bin/myapp --config /etc/myapp.conf (code=exited, status=203/EXEC)
   Main PID: 4821 (code=exited, status=203/EXEC)

Jul 21 14:03:11 host systemd[1]: myapp.service: Main process exited, code=exited, status=203/EXEC
Jul 21 14:03:11 host systemd[1]: myapp.service: Failed with result 'exit-code'.
```

Read the `Result:` value. It tells you the category of failure before you read a single application log:

- **exit-code:** the process ran and returned non-zero. Could be your app, could be systemd failing to exec it.
- **signal:** the kernel or something else killed it (SIGSEGV, SIGABRT).
- **timeout:** it didn't signal readiness within `TimeoutStartSec=`, 90 seconds by default. Almost always a `Type=` mismatch.
- **oom-kill:** the OOM killer took it. Check `MemoryMax=` and actual usage.

The `status=` number matters too. Anything in the 2xx range comes from systemd's own exec machinery, before your program ran:

| Status | Name | What to check |
|---|---|---|
| 200 | CHDIR | `WorkingDirectory=` missing or unreadable by the service user |
| 203 | EXEC | `ExecStart=` path wrong, not absolute, not executable, or bad shebang |
| 209 | STDOUT | `StandardOutput=` target can't be opened |
| 217 | USER | `User=` doesn't exist or can't be switched to |
| 226 | NAMESPACE | a sandboxing option (`ProtectSystem=`, `ReadWritePaths=`) points at a missing path |

## Read the journal

Status shows the tail. For the full story:

```bash
journalctl -u myapp.service -b --no-pager
journalctl -u myapp.service -f
```

`-b` scopes to the current boot, `--no-pager` dumps straight to stdout for grepping, and `-f` follows live while you poke at a restart loop. If the app logs to stdout/stderr, systemd captures all of it here, so this is usually where the real error message lives.

## The usual suspects

Once you know the Result and status code, it's pattern matching. These seven cover the vast majority of what we've seen.

**1. Bad ExecStart.** `203/EXEC` almost always means the path in `ExecStart=` is wrong or not executable. systemd resolves a bare name against a fixed compile-time search path, not your shell's `$PATH`, so give it the full path and confirm the bit is set:

```bash
ls -l /opt/myapp/bin/myapp
file /opt/myapp/bin/myapp     # ELF binary, or a script with a valid shebang?
chmod +x /opt/myapp/bin/myapp
```

A missing shebang or a script pointing at a `/usr/bin/python` that isn't there shows up as `203` too.

**2. Working directory or user.** `200/CHDIR` and `217/USER`, per the table. A service running as `www-data` can't `cd` into a root-only `/root/app`.

**3. Missing environment.** A missing `EnvironmentFile=` is fatal unless prefixed with a dash: `EnvironmentFile=-/etc/default/myapp` makes it optional. Confirm the file exists and the values look right.

**4. Type mismatch.** Your app forks and daemonizes itself, but the unit says `Type=simple`. systemd expects the main process to stay in the foreground, so when the parent exits it thinks the service died. If the app backgrounds itself, use `Type=forking`. If it supports `sd_notify`, use `Type=notify`, which gives systemd an accurate readiness signal. Most modern apps run in the foreground and want plain `Type=simple`.

**5. Dependencies and ordering.** The service started before the network, a database, or an NFS mount. `After=` controls ordering, `Requires=` or `Wants=` controls whether the dependency is pulled in. A service binding a specific IP wants `After=network-online.target` plus `Wants=network-online.target`. One that reads from a mount wants `RequiresMountsFor=`.

**6. Permissions and SELinux.** The binary runs, the user is right, and it still dies opening a file or socket. On SELinux systems check `ausearch -m avc -ts recent` for `AVC denied`. On plain DAC systems it's usually a file mode or a port under 1024 without `AmbientCapabilities=CAP_NET_BIND_SERVICE`.

**7. Syntax and stale state.** Verify the unit, and reload after every edit:

```bash
systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl restart myapp.service
```

`daemon-reload` is the one people forget: systemd keeps running the old in-memory copy until you reload.

## See what systemd actually loaded

`systemctl cat myapp.service` shows the effective unit including drop-ins. For a fix that survives package upgrades, use `sudo systemctl edit myapp.service`, which writes `override.conf` under `/etc/systemd/system/myapp.service.d/`:

```ini
# /etc/systemd/system/myapp.service.d/override.conf
[Service]
Type=simple
ExecStart=
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp.conf
WorkingDirectory=/opt/myapp
User=myapp
EnvironmentFile=-/etc/default/myapp
```

The empty `ExecStart=` first clears the inherited value; without it, systemd rejects a second `ExecStart` on a non-oneshot service.

## Questions people ask

### What does status=203/EXEC mean in systemctl?

It means systemd could not execute the program in `ExecStart=`, so your code never ran. The usual causes are a relative or wrong path, a file without the execute bit, a script with no shebang or a shebang pointing at a missing interpreter, or SELinux blocking a binary in an unusual directory. Check with `ls -l` and `file` on the exact path in the unit.

### How do I fix "start request repeated too quickly"?

That message is the rate limiter, not the root cause. By default systemd allows 5 starts within 10 seconds (`DefaultStartLimitBurst=5`, `DefaultStartLimitIntervalSec=10s`); a unit with `Restart=` that keeps crashing hits it and stops retrying. Find the real failure in `journalctl -u NAME`, fix it, then run `systemctl reset-failed NAME` and start the unit again.

### Why does my service work by hand but fail under systemd?

systemd starts it with a different environment: no login shell profile, a minimal `PATH`, the `User=` from the unit instead of you, and a working directory of `/` unless set. Reproduce it faithfully with `sudo -u myapp env -i /opt/myapp/bin/myapp --config /etc/myapp.conf`, and move any variables you rely on into `Environment=` or `EnvironmentFile=`.

## The call we'd make

Read `systemctl status` for the Result and exit code first, because a 2xx status means the unit file is wrong and you can skip the app logs entirely. Pull the journal for anything else, fix it with a drop-in rather than editing the vendor file, and always `daemon-reload` before you restart. Nine times out of ten it's a bad `ExecStart`, a `Type=` mismatch, or a missing dependency. For the wider playbook, see our [Linux troubleshooting guide](/blog/linux-troubleshooting-guide).

## Sources

- [systemd.exec(5): process exit codes and execution environment](https://man7.org/linux/man-pages/man5/systemd.exec.5.html)
- [systemd.unit(5): StartLimitIntervalSec, StartLimitBurst, RequiresMountsFor](https://man7.org/linux/man-pages/man5/systemd.unit.5.html)
- [systemd-system.conf(5): default start timeout and start-limit values](https://man7.org/linux/man-pages/man5/systemd-system.conf.5.html)
- [systemd.service(5): Type=, ExecStart= and Restart=](https://man7.org/linux/man-pages/man5/systemd.service.5.html)