# journald Log Management: Retention, Filtering, and Forwarding URL: https://www.devopsness.com/blog/journald-log-management-linux Author: Kiril Urbonas Published: 2026-08-29 Updated: 2026-08-29 Categories: Linux Tags: Linux, Observability, Systemd, Logging, Sysadmin Summary: journald is the default log sink on every systemd distro, and most of it runs on defaults nobody chose. Here is how to actually control it. --- # journald Log Management: Retention, Filtering, and Forwarding Every systemd-based distro ships with `journald` capturing stdout/stderr and structured metadata from every service, but most boxes run it entirely on defaults, including, on some distros, storing logs in `/run` so they vanish on every reboot. That surprises people exactly once, usually while trying to find what a service logged right before it crashed and rebooted the box. ## Check whether logs even survive a reboot ```bash $ journalctl --disk-usage Archived and active journals take up 96.0M in the file system. $ cat /etc/systemd/journald.conf | grep -i storage #Storage=auto ``` `Storage=auto` (the commented-out default) means: log to `/var/log/journal` if that directory exists, otherwise fall back to volatile `/run/log/journal`, which is wiped on every reboot. If `/var/log/journal` doesn't exist on your box, your journal history is one reboot away from gone: ```bash $ ls -d /var/log/journal 2>&1 ls: cannot access '/var/log/journal': No such file or directory ``` Fix it by creating the directory and restarting the journal: ```bash $ sudo mkdir -p /var/log/journal $ sudo systemd-tmpfiles --create --prefix /var/log/journal $ sudo systemctl restart systemd-journald ``` `journald` picks up persistent storage automatically once the directory exists. No config change is needed for that part, `Storage=auto` was already going to use it. ## Control how much disk it's allowed to eat Unbounded log growth is the other default surprise. Set explicit caps in `/etc/systemd/journald.conf`: ```ini # /etc/systemd/journald.conf [Journal] Storage=persistent SystemMaxUse=2G SystemKeepFree=1G SystemMaxFileSize=100M MaxRetentionSec=30day ``` - `SystemMaxUse`: hard cap on total journal disk usage. - `SystemKeepFree`: journald stops writing before it eats into this much free disk space, regardless of `SystemMaxUse`. This is the guard against journald itself causing a disk-full incident. - `MaxRetentionSec`: deletes entries older than this even if under the size cap; useful when you want a time-bounded retention policy rather than a size-bounded one. ```bash $ sudo systemctl restart systemd-journald $ journalctl --disk-usage ``` Without at least `SystemMaxUse` or `SystemKeepFree` set, a chatty service in a crash loop can fill the disk with journal entries faster than most people expect. This is the same failure mode as [disk full on Linux](/blog/fix-disk-full-linux), just journald-flavored. ## Filtering: journalctl beyond `-f` ```bash $ journalctl -u nginx --since "1 hour ago" $ journalctl -u nginx -p err # priority: only error and above $ journalctl -u nginx --since "2026-08-29 09:00" --until "2026-08-29 09:15" $ journalctl -k # kernel messages only (replaces dmesg) $ journalctl -f -u nginx -u postgresql # follow multiple units interleaved by time ``` Priority filtering (`-p`) accepts syslog levels (`emerg`, `alert`, `crit`, `err`, `warning`, `notice`, `info`, `debug`) and includes everything at or above the given level; `-p err` shows `err`, `crit`, `alert`, and `emerg`, not just exact matches. Structured field filtering is where journald earns its keep over flat text logs: every entry carries metadata you can query directly: ```bash $ journalctl _PID=12345 $ journalctl _SYSTEMD_UNIT=nginx.service _COMM=nginx $ journalctl -o json-pretty -u nginx -n 1 # see every field a unit's entries actually carry ``` That last command is the fastest way to discover what's queryable for a given service; every key in the JSON output is a valid filter field. ## Correlating a crash with what came before it ```bash $ systemctl status myapp # note the PID and the boot it crashed in, then: $ journalctl -u myapp -b -1 -n 100 # last 100 lines from the *previous* boot $ journalctl -u myapp --since "-10min" -p warning ``` `-b -1` is easy to forget and is exactly what you need after a crash-triggered reboot. `-b 0` (the default, current boot) shows nothing useful because the process that crashed isn't in this boot's journal at all. ## Forwarding to a centralized logging stack journald's own storage is local and rotates; for anything you need retained past that or searchable across a fleet, forward it. Two common paths: **Forward to traditional syslog** (if you already run rsyslog or syslog-ng, which then ships elsewhere): ```ini # /etc/systemd/journald.conf [Journal] ForwardToSyslog=yes ``` **Ship structured JSON directly** with `systemd-cat` or a log shipper reading the journal natively (Vector, Fluent Bit, and Promtail all support journald as a native input, avoiding a syslog translation step and keeping the structured fields intact): ```bash $ journalctl -o json --since "5 min ago" -f | your-shipper-of-choice ``` Reading the journal as JSON preserves fields like `_SYSTEMD_UNIT`, `_PID`, and `PRIORITY` as structured data in whatever downstream system receives it, instead of flattening everything back into a single text line the way syslog forwarding does. ## The fix in order 1. `journalctl --disk-usage` and check `Storage=`: confirm logs are persistent (`/var/log/journal` exists) before you need history that a reboot just erased. 2. Set `SystemMaxUse`/`SystemKeepFree` explicitly. Don't run without a disk cap, especially on any box where a crash-looping service could write logs faster than it fails. 3. Use `-u`, `-p`, `--since`/`--until`, and structured field filters (`_PID=`, `_SYSTEMD_UNIT=`) instead of piping through `grep`; journald's own filtering is faster and more precise. 4. After a crash-reboot, `journalctl -u -b -1`: the previous boot, not the current one. 5. Forward to a real logging stack for anything you need retained or searched across more than one host; native journald input (Vector, Fluent Bit, Promtail) keeps structured fields that syslog forwarding flattens away. This pairs with [systemd and modern Linux service management](/blog/systemd-and-modern-linux-service-management) for the service-management side of the same daemon, and [diagnosing disk full on Linux](/blog/fix-disk-full-linux) when journald itself turns out to be what filled the disk. ## The call we'd make Set `Storage=persistent` and an explicit `SystemMaxUse`/`SystemKeepFree` on every box on day one. It costs one config file and prevents both "logs vanished on reboot" and "journald filled the disk." Forward to a real logging stack the moment you need retention or cross-host search beyond what a single box's local journal can give you.