journald Log Management: Retention, Filtering, and Forwarding

Every systemd-based distro ships with journald capturing stdout/stderr and structured metadata from every service, but most boxes run it entirely on defaults, including, on some distros, storing logs in /run so they vanish on every reboot. That surprises people exactly once, usually while trying to find what a service logged right before it crashed and rebooted the box.

Check whether logs even survive a reboot

$ journalctl --disk-usage
Archived and active journals take up 96.0M in the file system.

$ cat /etc/systemd/journald.conf | grep -i storage
#Storage=auto

Storage=auto (the commented-out default) means: log to /var/log/journal if that directory exists, otherwise fall back to volatile /run/log/journal, which is wiped on every reboot. If /var/log/journal doesn't exist on your box, your journal history is one reboot away from gone:

$ ls -d /var/log/journal 2>&1
ls: cannot access '/var/log/journal': No such file or directory

Fix it by creating the directory and restarting the journal:

$ sudo mkdir -p /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal
$ sudo systemctl restart systemd-journald

journald picks up persistent storage automatically once the directory exists. No config change is needed for that part, Storage=auto was already going to use it.

Control how much disk it's allowed to eat

Unbounded log growth is the other default surprise. Set explicit caps in /etc/systemd/journald.conf:

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
SystemMaxFileSize=100M
MaxRetentionSec=30day

• SystemMaxUse: hard cap on total journal disk usage.
• SystemKeepFree: journald stops writing before it eats into this much free disk space, regardless of SystemMaxUse. This is the guard against journald itself causing a disk-full incident.
• MaxRetentionSec: deletes entries older than this even if under the size cap; useful when you want a time-bounded retention policy rather than a size-bounded one.

$ sudo systemctl restart systemd-journald
$ journalctl --disk-usage

Without at least SystemMaxUse or SystemKeepFree set, a chatty service in a crash loop can fill the disk with journal entries faster than most people expect. This is the same failure mode as disk full on Linux, just journald-flavored.

Filtering: journalctl beyond -f

$ journalctl -u nginx --since "1 hour ago"
$ journalctl -u nginx -p err                    # priority: only error and above
$ journalctl -u nginx --since "2026-08-29 09:00" --until "2026-08-29 09:15"
$ journalctl -k                                  # kernel messages only (replaces dmesg)
$ journalctl -f -u nginx -u postgresql           # follow multiple units interleaved by time

Priority filtering (-p) accepts syslog levels (emerg, alert, crit, err, warning, notice, info, debug) and includes everything at or above the given level; -p err shows err, crit, alert, and emerg, not just exact matches.

Structured field filtering is where journald earns its keep over flat text logs: every entry carries metadata you can query directly:

$ journalctl _PID=12345
$ journalctl _SYSTEMD_UNIT=nginx.service _COMM=nginx
$ journalctl -o json-pretty -u nginx -n 1        # see every field a unit's entries actually carry

That last command is the fastest way to discover what's queryable for a given service; every key in the JSON output is a valid filter field.

Correlating a crash with what came before it

$ systemctl status myapp
# note the PID and the boot it crashed in, then:
$ journalctl -u myapp -b -1 -n 100               # last 100 lines from the *previous* boot
$ journalctl -u myapp --since "-10min" -p warning

-b -1 is easy to forget and is exactly what you need after a crash-triggered reboot. -b 0 (the default, current boot) shows nothing useful because the process that crashed isn't in this boot's journal at all.

Forwarding to a centralized logging stack

journald's own storage is local and rotates; for anything you need retained past that or searchable across a fleet, forward it. Two common paths:

Forward to traditional syslog (if you already run rsyslog or syslog-ng, which then ships elsewhere):

# /etc/systemd/journald.conf
[Journal]
ForwardToSyslog=yes

Ship structured JSON directly with systemd-cat or a log shipper reading the journal natively (Vector, Fluent Bit, and Promtail all support journald as a native input, avoiding a syslog translation step and keeping the structured fields intact):

$ journalctl -o json --since "5 min ago" -f | your-shipper-of-choice

Reading the journal as JSON preserves fields like _SYSTEMD_UNIT, _PID, and PRIORITY as structured data in whatever downstream system receives it, instead of flattening everything back into a single text line the way syslog forwarding does.

The fix in order

1. journalctl --disk-usage and check Storage=: confirm logs are persistent (/var/log/journal exists) before you need history that a reboot just erased.
2. Set SystemMaxUse/SystemKeepFree explicitly. Don't run without a disk cap, especially on any box where a crash-looping service could write logs faster than it fails.
3. Use -u, -p, --since/--until, and structured field filters (_PID=, _SYSTEMD_UNIT=) instead of piping through grep; journald's own filtering is faster and more precise.
4. After a crash-reboot, journalctl -u <service> -b -1: the previous boot, not the current one.
5. Forward to a real logging stack for anything you need retained or searched across more than one host; native journald input (Vector, Fluent Bit, Promtail) keeps structured fields that syslog forwarding flattens away.

This pairs with systemd and modern Linux service management for the service-management side of the same daemon, and diagnosing disk full on Linux when journald itself turns out to be what filled the disk.

The call we'd make

Set Storage=persistent and an explicit SystemMaxUse/SystemKeepFree on every box on day one. It costs one config file and prevents both "logs vanished on reboot" and "journald filled the disk." Forward to a real logging stack the moment you need retention or cross-host search beyond what a single box's local journal can give you.