Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

systemd status=203/EXEC and 217/USER: Exec and Permission Failures

203 is a failed execve, 217 is a user systemd could not resolve. Both happen before your program runs, and the check that finds each takes one command.

~4 min readLinux
#Linux#systemd#Permissions#Troubleshooting
3D illustration of a service node with a padlock on its user badge and a broken exec arrow

Introduction & Quick Summary

Symptom. sudo systemctl start myapp fails immediately, and systemctl status myapp reports Active: failed (Result: exit-code) with Main process exited, code=exited, status=203/EXEC — or the service never starts because Failed to determine user credentials: No such process and status=217/USER. The unit file looks correct, journalctl -u myapp is empty because the process never ran, and testing the command by hand as root works.

Cause. Both codes are reported by systemd before your program runs. 203/EXEC means execve() on the ExecStart binary failed: the path is wrong or relative, the file is not executable, the shebang interpreter does not exist, the mount is noexec, or SELinux refuses the transition. 217/USER means systemd could not resolve the User= or Group= name at the moment of start — a typo, a missing system account, or a directory-service user that is not available yet at boot.

Fix. Resolve the exec path and the identity separately, as the service would. 203/EXEC: use an absolute path, confirm test -x, check the shebang with head -1, and verify the mount options. 217/USER: create the account (useradd --system), or order the unit after nss-user-lookup.target and network-online.target when the user comes from LDAP. systemd-analyze verify myapp.service catches most of both before you restart anything.

Core Mechanics, Signatures & Reproducible Setup

systemd forks, applies sandboxing, switches identity, then calls execve(). Each step has its own exit status, which is why the number in status= tells you exactly which step failed.

code
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml User=myapp Group=myapp WorkingDirectory=/var/lib/myapp

A minimal unit that reproduces both failures on any system:

ini
# /etc/systemd/system/breakme.service [Unit] Description=reproducer [Service] Type=simple User=ghostuser ExecStart=/opt/myapp/run.sh Restart=no [Install] WantedBy=multi-user.target
bash
sudo systemctl daemon-reload && sudo systemctl start breakme systemctl status breakme --no-pager # status=217/USER: ghostuser does not exist sudo useradd --system --no-create-home --shell /usr/sbin/nologin ghostuser sudo systemctl start breakme systemctl status breakme --no-pager # status=203/EXEC: /opt/myapp/run.sh missing

Reproduce it as the service user, which is the check that actually finds permission problems:

bash
sudo -u myapp /opt/myapp/run.sh # prints the real error, e.g. Permission denied namei -l /opt/myapp/run.sh # every path component needs o+x for the service user stat -c '%A %U:%G %n' /opt/myapp/run.sh

Key Comparison Table

systemd exit statuses name the failing step, not the program.

status=MeaningTypical causeCheck
200/CHDIRCould not change to WorkingDirectoryDirectory missing or unreadablenamei -l $WorkingDirectory
203/EXECexecve() failedMissing, relative, not exec, bad shebangsudo -u user /abs/path
216/GROUPGroup not foundGroup= typo or missing accountgetent group <name>
217/USERUser not foundUser= typo, NSS not ready at bootgetent passwd <name>
226/NAMESPACESandbox setup failedProtectSystem=, PrivateTmp= conflictsystemd-analyze security
233/RESOURCESResource limits refusedMemoryMax= below the runtime floorsystemctl show -p MemoryMax
1–255The program ran and exitedApplication errorjournalctl -u name -n 50

The two failures have almost no overlap in diagnosis:

Aspect203/EXEC217/USER
When it failsAfter identity switch, in execveBefore any process starts
Journal outputUsually nothing from your appFailed to determine user credentials
Affected by chmodYesNo
Affected by NSS/LDAPNoYes
Diagnose withsudo -u <user> <path>getent passwd <user>

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. A script with the wrong interpreter

A relative ExecStart=./run.sh is rejected outright; a script whose shebang names an interpreter that is not installed fails at execve with the same 203:

ini
ExecStart=/opt/myapp/run.sh # absolute, executable
bash
head -1 /opt/myapp/run.sh # #!/usr/bin/env bash -> needs env and bash on PATH

#!/usr/bin/env node fails on a minimal image without /usr/bin/env and depends on the PATH systemd gives the unit (a very short default, not your shell's). Name the interpreter explicitly: ExecStart=/usr/bin/node /opt/myapp/server.js.

2. CRLF line endings in the shebang

A file edited on Windows ends its first line with \r, so the kernel looks for an interpreter named /bin/bash\r and reports 203/EXEC (or bad interpreter: /bin/bash^M when run by hand):

bash
file /opt/myapp/run.sh # 'with CRLF line terminators' sed -i 's/\r$//' /opt/myapp/run.sh

The same edit also breaks EnvironmentFile= parsing when a value ends in a carriage return.

3. Permissions on the path, not just the file

test -x /opt/myapp/run.sh passes as root while the service user still cannot execute it, because a parent directory lacks the execute bit. Every component of the path needs o+x (or group access for the service account):

bash
namei -l /opt/myapp/run.sh # drwxr-x--- root root /opt <- blocks every non-root user chmod o+x /opt chmod 750 /opt/myapp && chown -R myapp:myapp /opt/myapp

4. noexec mounts and encrypted home directories

A binary on a noexec mount returns 203/EXEC even when everything looks right. This bites deployments under /home, /tmp (often noexec,nosuid) and on network shares:

bash
findmnt -no TARGET,OPTIONS /home /tmp /opt # /home ext4 rw,noexec,nosuid

Move the binary to /usr/local/bin or /opt, or drop noexec from the mount. For an encrypted home that is not mounted at boot, add RequiresMountsFor=/home/myapp so systemd waits for it.

5. Users that exist only after the network is up

Directory-service accounts resolve asynchronously at boot. A unit that starts early fails with 217/USER while getent passwd myapp works interactively minutes later:

ini
[Unit] After=network-online.target nss-user-lookup.target Wants=network-online.target

nss-user-lookup.target is the ordering point systemd provides exactly for this. If the host is offline, systemd's own NSS fallback flags (systemd-remount-fs, nss-systemd) can also hide the account until the cache is warm.

6. Sandboxing options that collide with the executable

Hardening can turn a working unit into 203 or 226: ProtectSystem=strict makes /usr read-only (fine) but also hides paths under ReadWritePaths (not); PrivateTmp= gives the service its own /tmp; ProtectHome=true hides /home entirely; NoNewPrivileges= blocks a setuid helper the app spawns.

bash
systemd-analyze verify /etc/systemd/system/myapp.service systemd-analyze security myapp.service | tail -5 sudo systemctl show myapp -p ProtectHome -p ProtectSystem -p PrivateTmp

If a service worked before hardening was added, revert one directive at a time instead of guessing; the status changes between 203, 226 and a plain application failure depending on which path the option hides.

Common Compilation/Runtime Mistakes & Version Gotchas

  • Editing the unit without daemon-reload. systemd keeps the parsed unit in memory; systemctl daemon-reload is required after every change, and systemctl restart alone applies the old version.
  • % in ExecStart. systemd expands specifiers, so ExecStart=/bin/date +%Y fails; write %%Y. This is one of the few cases where a correct command breaks only under systemd.
  • Environment= with shell quoting. Environment="PATH=/opt/bin:$PATH" does not expand $PATH; expansion happens only for $VAR words in ExecStart, and only for variables the unit defines.
  • Restart=always hiding the cause. A unit in a restart loop buries the first failure. systemctl reset-failed myapp clears the counter, and journalctl -u myapp -b keeps the first attempt.
  • Type=forking without a PID file. systemd reports success even when the daemon dies immediately; Type=simple plus Restart=on-failure surfaces the crash instead.
  • PermissionsStartOnly= from old guides. It was removed; the modern equivalent is the + prefix on a single ExecStartPre=+/bin/chown … line.
  • SELinux and AppArmor. A binary in a home directory runs by hand and fails with 203 under systemd because the label is wrong: ls -Z /opt/myapp, then restorecon -Rv /opt/myapp (RHEL family) instead of setenforce 0.

Related tools. When two unit versions disagree about a path or an environment block, paste them into the text diff tool to see the difference that matters; when a service fails while reading a JSON config, format it with the JSON formatter and check the file the unit actually points at, since a WorkingDirectory= change quietly changes which relative config gets loaded.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.