systemd-journald.service and journalctl

systemd-journald.service - It collects information from different sources and loads the messages into the journal.

The systemd journal is not a large text file. It’s a binary file maintained by the daemon. So, it can’t be opened with a text editor. We use journalctl command to view  systemd journal

The systemd journal is stored in /run/log by default and its content are cleared after reboot.

journalctl command shows full system journal, starting with oldest log entty

[root@ServerX~]# journalctl
-- Logs begin at Wed 2019-12-11 04:25:00 AEDT, end at Wed 2019-12-11 07:24:56 AEDT. --
Dec 11 04:25:00 localhost systemd-journal[95]: Runtime journal is using 8.0M (max allowed 756.4M, trying to leave 1.1G free of
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpuset
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpu
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpuacct
Dec 11 04:25:00 localhost kernel: Linux version 3.10.0-693.5.2.el7.x86_64 (mockbuild@x86-041.build.eng.bos.redhat.com) (gcc ve
Dec 11 04:25:00 localhost kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.5.2.el7.x86_64 root=UUID=3ed41454-00c8-480
Dec 11 04:25:00 localhost kernel: e820: BIOS-provided physical RAM map:
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved

Only list log entry of priority err or above

[root@ServerX~]# journalctl -p err
-- Logs begin at Wed 2019-12-11 04:25:00 AEDT, end at Wed 2019-12-11 07:28:30 AEDT. --
Dec 11 04:25:00 localhost kernel: Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Dec 11 04:25:00 localhost kernel: Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Dec 11 04:25:01 localhost iscsid[384]: iSCSI daemon with pid=385 started!
Dec 11 04:25:02 localhost iscsid[384]: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
Dec 11 04:25:02 localhost iscsid[384]: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not c
Dec 11 04:25:02 localhost iscsid[384]: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
Dec 11 04:25:02 localhost iscsid[384]: can't open iscsid.safe_logout configuration file /etc/iscsi/iscsid.conf

In addition to log , there are fields attached to the log entries that can only be seen when verbose output is turned on.

All these filed can be used to filter the output of journal query. some import fields

_COMM=sshd	The name of the command
_EXE=/usr/sbin/sshd	The path of the executable for the process
_UID=0	UID of the user running the process
_PID=2123	PID of the process
_SYSTEMD_UNIT=sshd.service	systemd unit that started the process

[root@ServerX~]# journalctl -o verbose
Wed 2019-12-11 04:25:37.940589 AEDT [s=4baf2122c2ee451ea4f5f05ae0dba467;i=566;b=313dc93a97174097905f360bcb417d8e;m=259060a;t=5
    PRIORITY=6
    _UID=0
    _GID=0
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=313dc93a97174097905f360bcb417d8e
    _MACHINE_ID=72e8116c885b46de947ad2ca3d0eba76
    _HOSTNAME=ServerX
    _CAP_EFFECTIVE=1fffffffff
    _TRANSPORT=syslog
    SYSLOG_FACILITY=10
    SYSLOG_IDENTIFIER=sshd
    _COMM=sshd
    _EXE=/usr/sbin/sshd
    _SYSTEMD_CGROUP=/system.slice/sshd.service
    _SYSTEMD_UNIT=sshd.service
    SYSLOG_PID=2123
    MESSAGE=Accepted publickey for ec2-user from 10.0.0.1 port 38186 ssh2: RSA SHA256:GvmOX7imV2RQzRTZ/ojY9jEgf0PeboTkwKx
    _PID=2123
    _CMDLINE=sshd: ec2-user [priv
    _SOURCE_REALTIME_TIMESTAMP=1575998737940589

Some of the important command

journalctl -n - shows last 10 log entries

journalctl -n 5 - shows last 5 log entries

journalctl -p err - Only list log entry of priority err or above
journalctl -u sshd - Show messages for the specified systemd unit UNIT 

journalctl -f - Outputs last 10 lines of journal and continue to output new journal entries like tail -f

journalctl --since today - Shows all log entries that got recorded today

journalctl --since "2012-12-30 20:30:00" --unitl "2010-12-31 12:00:00" - output jounal entry between these 2 dates. Date format YYYY-MM-DD hh:mm:ss. 

jounralctl -o verbose - to show additional field attached to the log entry
journalctl _SYSTEMD_UNIT=sshd.service _PID=2123 - filtering with additional field