Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Linux systemd: Managing Services with systemctl, Unit Files, and journalctl

Unit file sections, Type= semantics, restart policies and the daemon-reload step everyone forgets, plus the journal queries that explain the failure.

~4 min readLinux
#Linux#systemd#SysAdmin
3D illustration of a systemd PID 1 supervisor managing service unit lifecycles and structured journald log streams

Quick Summary

systemd is PID 1 and the service manager on nearly every modern distribution. It reads unit files that describe what to start, how to start it, what it depends on, and what to do when it exits; systemctl controls those units, and journalctl reads the logs they produce. The three commands cover most administration: systemctl status to see what a service is doing, systemctl restart to change that, journalctl -u to find out why it failed.

A unit file is three sections. [Unit] holds description and ordering, [Service] holds the command line, user, restart policy, and sandboxing, [Install] holds which targets enable it. The part people miss is that WantedBy=multi-user.target in [Install] only takes effect when you run systemctl enable; a unit file dropped into /etc/systemd/system/ is known to systemd but not started at boot until it is enabled.

The second thing people miss is that systemd does not re-read unit files automatically. Editing a unit and running restart runs the old definition until systemctl daemon-reload. Drop-in overrides under NAME.service.d/*.conf are the safe way to change a vendor unit, because a package upgrade overwrites the unit file itself.

Basic Syntax & Signature

ini
# /etc/systemd/system/api.service [Unit] Description=Backend API Documentation=https://example.com/docs After=network-online.target postgresql.service Wants=network-online.target Requires=postgresql.service [Service] Type=simple User=api Group=api WorkingDirectory=/srv/api EnvironmentFile=/etc/api/env Environment=NODE_ENV=production PORT=8080 ExecStart=/usr/bin/node /srv/api/server.js ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=3 TimeoutStopSec=20 KillSignal=SIGTERM LimitNOFILE=65535 StandardOutput=journal StandardError=journal SyslogIdentifier=api # Hardening: each line removes an attack surface, at the cost of a feature. NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=true ReadWritePaths=/srv/api/uploads [Install] WantedBy=multi-user.target
bash
sudo systemctl daemon-reload # after editing any unit file sudo systemctl enable --now api.service # start now and at boot systemctl status api.service # state, recent log lines, main PID sudo systemctl restart api.service sudo systemctl reload-or-restart api.service systemctl is-enabled api.service systemctl list-units --type=service --state=failed journalctl -u api.service -n 100 --no-pager journalctl -u api.service -f # follow journalctl -u api.service --since "10 min ago" journalctl -u api.service -p err -b # errors since the last boot journalctl -u api.service -o json-pretty | head

Type= is the decision that determines when systemd considers the service started: simple (default) starts and forks immediately, exec waits until execve succeeds, forking waits for the parent to exit, notify waits for sd_notify(READY=1), oneshot waits for the command to finish, and idle waits until the job queue is empty.

Key Comparison Table

TaskCommandNotes
Start nowsystemctl start apiNot persistent
Start at bootsystemctl enable apiReads [Install]
Bothsystemctl enable --now apiOne transaction
Stopsystemctl stop apiRuns ExecStop
Restartsystemctl restart apiStop then start
Reload configsystemctl reload apiNeeds ExecReload
Re-read unitssystemctl daemon-reloadRequired after edits
Show unit filesystemctl cat apiIncludes drop-ins
Override safelysystemctl edit apiWrites a drop-in
Prevent startsystemctl mask apiSymlink to /dev/null
Undo masksystemctl unmask apiRestores the unit
Boot logsjournalctl -b-b -1 for the previous boot
Which unit owns a processsystemctl status <PID>Or cat /proc/<PID>/cgroup
List timerssystemctl list-timersCron replacements
Check failure causesystemctl status api / journalctl -u api -p errExit code line

Practical Edge Cases & Deep Dives

1. A service that starts fine by hand and fails under systemd

bash
# Works in a shell: /srv/api/server.js # Fails as a unit: PATH, home, cwd, and env are minimal under systemd. systemctl status api # ... status=203/EXEC

203/EXEC means the kernel could not execute ExecStart. Common causes: a relative path (systemd requires an absolute path), a missing shebang, an interpreter that lives in /usr/local/bin not on systemd's PATH, or an SELinux label. Fix with an absolute interpreter path and an explicit environment:

ini
[Service] Environment=PATH=/usr/local/bin:/usr/bin:/bin ExecStart=/usr/bin/node /srv/api/server.js

Services do not inherit your shell, your ~/.bashrc, or your working directory. WorkingDirectory= and EnvironmentFile= exist to make that explicit.

2. Restart loops and the rate limit

bash
systemctl status api # Active: activating (auto-restart) (Result=exit-code) # ... start request repeated too quickly

Restart=on-failure with RestartSec=1 will spin forever, and systemd eventually refuses to start the unit after five attempts in ten seconds. Two adjustments are usually needed: a longer RestartSec, and StartLimitIntervalSec/StartLimitBurst in [Unit]:

ini
[Unit] StartLimitIntervalSec=60 StartLimitBurst=5 [Service] Restart=always RestartSec=5

Use Restart=always for daemons that should never be down and on-failure for ones that exit cleanly when there is nothing to do. Restart=no on a oneshot unit that is supposed to run once.

3. Graceful shutdown that respects the timeout

ini
[Service] KillSignal=SIGTERM TimeoutStopSec=30 KillMode=mixed

systemctl stop sends SIGTERM to the main process by default and waits TimeoutStopSec before sending SIGKILL. If the application drains connections in its SIGTERM handler it needs those thirty seconds; if it ignores the signal, systemctl stop appears to hang for exactly the timeout. KillMode=mixed sends the first signal to the main process only and the final SIGKILL to the whole cgroup, which avoids killing worker children before the parent has drained them.

4. Reload without dropping connections

ini
[Service] ExecReload=/bin/kill -HUP $MAINPID
bash
sudo systemctl reload nginx # sends HUP only if ExecReload exists sudo systemctl reload nginx || sudo systemctl restart nginx

reload fails with "Job type reload is not applicable" when the unit has no ExecReload. Wrapping reload with a restart fallback in deployment scripts is correct as long as the restart is the fallback, not the default — a restart drops in-flight requests that a reload would have preserved.

5. Drop-in overrides instead of editing vendor units

bash
sudo systemctl edit api # creates /etc/systemd/system/api.service.d/override.conf # override.conf [Service] Environment=LOG_LEVEL=debug LimitNOFILE=131072 sudo systemctl daemon-reload sudo systemctl restart api

A drop-in appends to or replaces specific keys without touching the packaged file, so apt upgrade does not silently revert the change. systemctl cat api shows the original followed by each drop-in in order, which is the fastest way to see the effective configuration.

6. Watching a unit fail at boot

bash
journalctl -b -u api.service --no-pager journalctl -b -1 -u api.service --no-pager # previous boot systemd-analyze verify /etc/systemd/system/api.service systemd-analyze critical-chain api.service

systemd-analyze verify catches syntax errors, unknown keys, and missing executables before a reboot. critical-chain shows which dependency delayed the unit, which is how an After=network-online.target that never becomes ready is identified — the service is not broken, it is waiting.

Common Compilation/Runtime Mistakes

SymptomCauseFix
Edits have no effectUnit file not re-readsystemctl daemon-reload
Service not starting at bootNot enabledsystemctl enable api
status=203/EXECBad ExecStart path or interpreterAbsolute paths, check shebang
status=200/CHDIRWorkingDirectory missingCreate it or point elsewhere
Starts by hand, fails as a unitEnvironment not inheritedEnvironment=, EnvironmentFile=
Restart loopRestart=always with a fast crashRestartSec, StartLimitBurst
Logs empty in journalctlOutput not journaledRemove StandardOutput=null, check SyslogIdentifier
Unit is maskedsystemctl mask was usedsystemctl unmask
Stop takes 90 secondsSIGTERM ignoredFix the handler, tune TimeoutStopSec
Config changes lost after upgradeVendor unit edited in placeUse systemctl edit drop-ins
Service killed mid-requestKillMode=control-group with workersKillMode=mixed plus a SIGTERM handler
Unit fails only in a containerNo systemd or no cgroup write accessRun systemd as PID 1 in the container

The pattern behind almost every one of these is that systemd runs services with less context than a shell does — no login environment, no terminal, a fixed PATH, and its own cgroup for killing. Writing the unit with that in mind, absolute paths and an explicit environment, is what turns a service that works "on my machine" into one that survives a reboot and a deploy.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.