Sunday, December 15, 2019

Controlling services using systemctl

Introduction to systemd

System startup and server processes are managed by the systemd system and Service Manager.This program provide a mentod for activating system resources, Server Daemons and other processes, both on boot time and on  running system.

Daemons are process that wait or run in the background performing various task.
To listen for connection Daemon uses a socket.

A Service often refers to one or more daemons,but starting or stopping a service may instead make a on-time change to the state of the system for example, to configure Network interfaces, which does not involve leaving a daemon process running afterward.

In RHEL 7 , Process ID 1 is systemd, the new init system.
In RHEL 6 and older system , Process ID 1 is init process

Few new feature of systemd
  • Paralleization capabilities, which increase the boot speed of a system
  • On-Demand starting of Daemons without requiring a separate service.
  • Automatic service dependency management prevents long timeouts, such as not starting a network service when the network is not active.
  • A method of tracking related processes together using Linux Control groups.

systemctl and systemd units
The systemctl command is used to manage different type of systemd objects, called units. A list of available unit type can be displayed with systemctl -t help .
[root@server1 ~]# systemctl -t help
Available unit types:
service
socket
busname
target
snapshot
device
mount
automount
swap
timer
path
slice
scope

Some of the common unit type are listed as follows.
service unit have a .service extension and represent system services. This type of unit is used to start  frequently accessed daemons, such as a web server
socket unit have a .socket extension and represent Inter process communication sockets. Control of the socket will be passed to a daemon or newly started service when a client connection is made.
Path unit have .path extension and are used to delay the activation of a service unitl a specific file system change occurs.This commonly used for service which use spool directories, such as a printing systems.


Command                                   
Task                                                                                
systemctl  status UNIT
View detailed information about a unit state
systemctl   stop UNIT
Stop a service ona running system
systemctl   start UNIT
Start a service on a running system
systemctl   restart UNIT
Restart a service on a running system
systemctl   reload UNIT
Reload configuration file of a running service.
systemctl   mask UNIT
Completely disable a service from being started, both manually and at boot.
systemctl   unmask UNIT
Make a masked service available
systemctl   enable UNIT
Configure a service to start at boot time
systemctl   disable UNIT
Disable a service from starting at boot time
systemctl   list-dependencies UNIT
List units which are required and wanted by the specified unit.
systemctl   is-active UNIT
Check whether the unit is in active state
systemctl   is-enabled UNIT
Check whether the unit is enabled to  start automatically at boot time

Checking Service status
[root@server1 ~]# systemctl status sshd.service
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-12-16 16:17:07 IST; 4h 7min left
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 1164 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1303 (sshd)
   CGroup: /system.slice/sshd.service
           └─1303 /usr/sbin/sshd

Dec 16 16:17:06 localhost.localdomain systemd[1]: Starting OpenSSH server daemon...
Dec 16 16:17:07 localhost.localdomain systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
Dec 16 16:17:07 localhost.localdomain sshd[1303]: Server listening on 0.0.0.0 port 22.
Dec 16 16:17:07 localhost.localdomain sshd[1303]: Server listening on :: port 22.
Dec 16 16:17:07 localhost.localdomain systemd[1]: Started OpenSSH server daemon.

Keyword:
Description:
loaded
Unit configuration file has been processed
active(running)
Running with one or more continuing processes
active(exited)
Sucessfully completed one-time configuration
active(waiting)
Running but waiting for a event
inactive
Not running
enabled
Will be started at boot time
disabled
Will not be started at boot time
static
Cannot be enabled, but may be started by an enabled unit.

List Unit files with systemctl

Query the state of all units to verify a system startup.
Both the command give the same output.
[root@Server1 ~]# systemctl list-units 
[root@Server1 ~]# systemctl 
UNIT                               LOAD   ACTIVE SUB     DESCRIPTION
auditd.service                     loaded active running Security Auditing Service
NetworkManager.service             loaded active running Network Manager

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.


Query the status of only the service units
[root@Server1 ~]# systemctl list-units --type=service
[root@Server1 ~]# systemctl --type=service

The --all option will add inactive units as well.
[root@Server1 ~]# systemctl list-units  -t service -all
[root@Server1 ~]# systemctl -t service -all
  UNIT                                   LOAD      ACTIVE   SUB     DESCRIPTION
  auditd.service                         loaded    active   running Security Auditing Service
  auth-rpcgss-module.service             loaded    inactive dead    Kernel Module supporting RPCSEC_GSS

List only failed services
[root@Server1 ~]# systemctl --failed --type=service
0 loaded units listed. Pass --all to see loaded but inactive units, too.

Viewing the enabled and disabled setting for all units. Optionally, Limit the type of unit.
[root@Server1 ~]# systemctl list-unit-files --type=service
UNIT FILE                                   STATE
arp-ethers.service                          disabled
auditd.service                              enabled
auth-rpcgss-module.service                  static

We can also check whether the particular service enabled to start after reboot using is-enabled option
[root@Server1 ~]# systemctl is-enabled sshd
enabled

We can check whether particular unit is active using is-active option
[root@Server1 ~]# systemctl is-active sshd
active


Unit Dependencies
Service may be started as dependencies of other services. If a socket unit is enabled and the service unit with the same name is not, the service will automatically started when a request is made to that socket unit.

[root@Server1 ~]# systemctl list-dependencies multi-user.target
multi-user.target
● ├─auditd.service
● ├─besclient.service
● ├─brandbot.path
● ├─choose_repo.service

--reverse show particular unit is required for which other units.
[root@Server1 ~]# systemctl list-dependencies multi-user.target --reverse
multi-user.target
● └─graphical.target


Masking services
A system may have conflicting services installed for a certain function, such as firewalls (iptables and firewalld). To prevent an administrator from accidentally starting a service, a service may be masked.
[root@Server1 ~]# systemctl mask iptables
ln -s '/dev/null' '/etc/systemd/system/iptables.service'
[root@Server1 ~]# systemctl unmask iptables
rm '/etc/systemd/system/iptables.service'

Enabling system daemons to start or stop at boot
Service are started at boot when links are created in the appropriate systemd configuration directories.
[root@server1 ~]# systemctl enable sshd
Created symlink from /etc/systemd/system/multi-user.target.wants/sshd.service to /usr/lib/systemd/system/sshd.service.

Same way disable can be used to disable the service from starting during the boot.
[root@server1 ~]# systemctl disable sshd
Removed symlink /etc/systemd/system/multi-user.target.wants/sshd.service.


Attaching system to subscription for software update in RHEL 7

Red Hat Subscription Management

Red Hat subscription management provides tools that can be used to entitle machines to product subscriptions.

There are Four basic tasks performed with Red Hat subscription manager 

Register a system to associate that system to Red Hat account.
Subscribe a system to entitle it to updated for selected Red Hat Products.
                   Subscription have special level of support,Expiration dates and default repository
                    The tool can be used wither to auto-attach or select a specific entitlement.
Enable Repositories to provide software packages. Multiples repository are enabled by default with each subscription, but other repositories such as updates or source code can be enabled or disabled as needed.

Review and track entitlement which are available or consumed. Subscription information can be viewed locally on a specific system or, for an account , in either the Red Hat customer portal Subscriptions page.

subscription-manager 

Use command subscription-manager to register a system in CLI. 

subscription-manager register - Register a system to Red Hat account
subscription-manager list --available - View available subscription
subscription-manager attach --auto - Auto-attach a subscription
subscription-manager list --consumed - View consumed subscription
subscription-manager repos --enable rhel-7-server-optional-source-rpms  -- Attach a repos
subscription-manager unregister  - unregister a system

Register a system to a Red Hat account
[root@server1 ~]# subscription-manager register
Registering to: subscription.rhsm.redhat.com:443/subscription
Username: abcxyz@gmail.com
Password:
The system has been registered with ID: d7ab5b88-25c1-441c-a585-c1befe53d289

View available subscription 
[root@server1 ~]# subscription-manager list --available
Subscription Name:   Red Hat Beta Access
Provides:            Red Hat CodeReady Linux Builder for x86_64 Beta
....
                     Red Hat Enterprise Linux for x86_64 Beta
                     Red Hat Enterprise Linux for Real Time for NFV Beta
.....
SKU:                 RH00069
Contract:            11844216
Pool ID:             8a85f98f686222970168759bd81e1929
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Level:       Self-Support
Service Type:        L1-L3
Subscription Type:   Standard
Ends:                01/22/2020
System Type:         Physical

Auto-attach a subscription 
[root@server1 ~]# subscription-manager attach --auto


Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed


View consumed subscription
[root@server1 ~]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat Developer Subscription
Provides:            Red Hat Enterprise Linux High Availability - Update Services for SAP Solutions
...
                     Red Hat Developer Tools Beta (for RHEL Server)
                     Red Hat Developer Toolset (for RHEL Server)
                     Red Hat Enterprise Linux for x86_64
....
SKU:                 RH00798
Contract:
Account:
Serial:              7529774378551146176
Pool ID:             8a85f99a684d00130168757a42a10509
Provides Management: No
Active:              True
Quantity Used:       1
Service Level:       Self-Support
Service Type:
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              01/22/2019
Ends:                01/22/2020
System Type:         Physical

Enable a repo
[root@server1 ~]# subscription-manager repos --enable rhel-7-server-optional-source-rpms

Repository 'rhel-7-server-optional-source-rpms' is enabled for this system.
[root@server1 ~]# yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
repo id                                                          repo name                                                                                        status
cd                                                               cd                                                                                                4,751
rhel-7-server-optional-source-rpms/7Server/x86_64                Red Hat Enterprise Linux 7 Server - Optional (Source RPMs)                                        3,185
rhel-7-server-rpms/7Server/x86_64                                Red Hat Enterprise Linux 7 Server (RPMs)                                                         26,733
rhel-rs-for-rhel-7-server-fastrack-rpms/x86_64                   Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Server) - Fastrack (RPMs)                      0

Unregister a system
[root@server1 ~]# subscription-manager unregister
System has been unregistered.


Entitlement Certificates
An Entitlement is a subscription that's been attached to a system.
Digital certificate are used to store current information about entitle on local system.
Once registered entitlement certificate are stored in /etc/pki and its sub-folder.

/etc/pki/product contains certificates which indicate Red Hat Products installed on the system
/etc/pki/consumer contains certificates which indicate the Red Hat account to which the system is registered.
/etc/pki/entitlement contains certificate which indicate which subscriptions are attached to the system.


The certificate can be inspected using rct utiliaty
[root@server1 ~]# rct stat-cert /etc/pki/entitlement/8775838808487347342.pem
Type: Entitlement Certificate
Version: 3.4
DER size: 3338b
Subject Key ID size: 20b
Content sets: 1071




Synchronize Files and Folders with rsync

rsync used to synchronize files between source and destination. Main advantage of rsync is it only copies the modified files .

Important option of rsync.
-n, --dry-run  - Perform a dry run,simulation of what happens when the command really get executed
-a , --archive   archive mode; equals -rlptgoD (no -H,-A,-X)
                   -r, --recursive           recurse into directories
                   -l, --links                 copy symlinks as symlinks
                   -p, --perms               preserve permissions
                   -t, --times                 preserve modification times
                   -g, --group               preserve group
                   -o, --owner              preserve owner (super-user only)
                   -D                            Synchronize device files
-v, --verbose               increase verbosity

-a archive mode doesn't add the following option which we need to include separately if required.
                -H, --hard-links            preserve hard links
                -A, --acls                      preserve ACLs (implies -p)
                -X, --xattrs                   preserve extended attributes, SELinux context

Important
               While entering the source directory of rsync, it is important to remember that whether tailing slash(/) is present or not.
               If  tailing slash (/) is there only the content of the directory will be copied

with tailing slash (/) , content of etc folder will be copied
[root@server1 ~]# rsync -av /etc/ /tmp
...
yum/pluginconf.d/langpacks.conf.rpmnew
yum/pluginconf.d/product-id.conf
..
sent 86256 bytes  received 11501 bytes  195514.00 bytes/sec

With-out tailing slash (/) , etc folder will also be copied
[root@server1 ~]# rsync -av /etc /tmp
..
etc/yum/pluginconf.d/langpacks.conf
etc/yum/pluginconf.d/langpacks.conf.rpmnew
etc/yum/pluginconf.d/product-id.conf
...
sent 86256 bytes  received 11501 bytes  195514.00 bytes/sec 







Configuring Host names and Name Resolution in RHEL 7

Changing the system Host Name

A static Host Name can be specified in the /etc/hostname .
Older version of RHEL(RHEL 6 and previous version) stores host name in /etc/sysconfig/network

The hostnamectl command is used to modify the file /etc/hostname . 
If the file doesn't exist hostname is set by Reverse DNS query once the interface has an IP address.
[root@rhel7-server1 ~]# hostnamectl set-hostname server1.example.com
[root@rhel7-server1 ~]# hostnamectl status
   Static hostname: server1.example.com
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 6208cf32c49449969e11e369a02b77d8
           Boot ID: 7e4ae6a0fa4d4e989928c0d2b40b7205
    Virtualization: kvm
  Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.3:GA:server
            Kernel: Linux 3.10.0-514.el7.x86_64
      Architecture: x86-64
[root@rhel7-server1 ~]# cat /etc/hostname
server1.example.com

Configure Name Resolution

The stub resolver is used to convert host names to IP addresses or the reverse.

The Stub resolver check the following things for Name Resolution in the the order
  1. /etc/hosts are checked first
  2. /etc/resolv.conf , If the entry is not found in /etc/hosts file, Stub resolver looks for DNS nameservers in /etc/resolv.conf
[root@server1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search example.com
nameserver 192.168.18.1

nameserver The IP address of a nameserver to query. 
Up to three nameserver directive may be given to provide backups if one is down.

search a list of domain names to try with a short host names.

NetworkManager will update the /etc/resolve.conf file using DNS setting in the connection configuration files. Use the nmcli to modify the connection.

[root@server1 ~]# nmcli connection modify "default" +ipv4.dns 8.8.8.8

[root@server1 ~]# nmcli connection up "default"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)

[root@server1 ~]# nmcli connection show "default" | grep IP4.DNS
IP4.DNS[1]:                             192.168.18.1
IP4.DNS[2]:                             8.8.8.8

[root@server1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search example.com
nameserver 192.168.18.1
nameserver 8.8.8.8

The default behavior of nmcli con mod ipv4.dns IP is to replace any previus DNS setting with the new IP list provided.
A +/- symbol in front of the ipv4.dns  argument will add or remove an individual entry.

getent hosts server1 - Used to test name resolution with the /etc/hosts
host server1.example.com - used to resolve name using DNS server

Important
If DHCP is in use, /etc/resolv.conf is automatically rewritten as interface are started. Unless you specify PEERDNS=no in the relevant interface configuration files.
This can be change can be made with nmcli

[root@server1 ~]# nmcli connection modify "default" ipv4.ignore-auto-dns yes




Wednesday, December 11, 2019

Maintain Time in RHEL 7 using chronyd

Setting time zone using timedatectl

The timedatectl command shows time related setting like current time,time zone and NTP synchronization setting of the system

[student@ServerX ~]$ timedatectl
      Local time: Thu 2019-12-12 06:11:34 AEDT
  Universal time: Wed 2019-12-11 19:11:34 UTC
        RTC time: Wed 2019-12-11 19:11:34
       Time zone: Australia/Sydney (AEDT, +1100)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2019-10-06 01:59:59 AEST
                  Sun 2019-10-06 03:00:00 AEDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2020-04-05 02:59:59 AEDT
                  Sun 2020-04-05 02:00:00 AEST

List time zones using the command "timedatectl list-timezones"
[student@ServerX ~]$ timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
Africa/Bangui
Africa/Banjul
Africa/Bissau

Set Time Zone using the command "timedatectl set-timezone Asia/Kolkata"

[root@ServerX ~]$ timedatectl set-timezone Asia/Kolkata


tzselect command is useful for identifying correct time zone names.It interactively prompts the user with question about the system's location and output the name of the correct time zone.
It doesn't make any changes to the time zone setting of the system.

[student@ServerX ~]$ tzselect
Please identify a location so that time zone rules can be set correctly.
Please select a continent or ocean.
 1) Africa
 2) Americas
 3) Antarctica
 4) Arctic Ocean
 5) Asia
 6) Atlantic Ocean
 7) Australia
 8) Europe
 9) Indian Ocean
10) Pacific Ocean
11) none - I want to specify the time zone using the Posix TZ format.
#? 5
Please select a country.
 1) Afghanistan           18) Israel                35) Palestine
 2) Armenia               19) Japan                 36) Philippines
 3) Azerbaijan            20) Jordan                37) Qatar
 4) Bahrain               21) Kazakhstan            38) Russia
 5) Bangladesh            22) Korea (North)         39) Saudi Arabia
 6) Bhutan                23) Korea (South)         40) Singapore
 7) Brunei                24) Kuwait                41) Sri Lanka
 8) Cambodia              25) Kyrgyzstan            42) Syria
 9) China                 26) Laos                  43) Taiwan
10) Cyprus                27) Lebanon               44) Tajikistan
11) East Timor            28) Macau                 45) Thailand
12) Georgia               29) Malaysia              46) Turkmenistan
13) Hong Kong             30) Mongolia              47) United Arab Emirates
14) India                 31) Myanmar (Burma)       48) Uzbekistan
15) Indonesia             32) Nepal                 49) Vietnam
16) Iran                  33) Oman                  50) Yemen
17) Iraq                  34) Pakistan
#? 14

The following information has been given:

        India

Therefore TZ='Asia/Kolkata' will be used.
Local time is now:      Thu Dec 12 00:50:32 IST 2019.
Universal Time is now:  Wed Dec 11 19:20:32 UTC 2019.
Is the above information OK?
1) Yes
2) No
#? 1

You can make this change permanent for yourself by appending the line
        TZ='Asia/Kolkata'; export TZ
to the file '.profile' in your home directory; then log out and log in again.

Here is that TZ value again, this time on standard output so that you
can use the /usr/bin/tzselect command in shell scripts:
Asia/Kolkata

Setting date and time using timedatectl

set-time option of timedatectl is used to set the time manually.
The time may be specified in the format "2012-10-30 18:17:16"

[root@ServerX ~]# timedatectl set-time 2012-10-30 18:17:16

Enabling NTP Synchronization 

The set-ntp option enabled or disables NTP Synchronization for automatic time adjustment.

[root@ServerX ~]# timedatectl set-ntp true

chronyd daemon

chronyd daemon is used to synchronize time from the NTP pool project in RHEL7. 
ntpd daemon is used to  synchronize time in RHEL 5 and RHEL 6 and older version.

The quality of NTP time source is determined by the stratum value reported by the time source.
The stratum determines the number of hops the machine is away from the high performance reference clock.

High performance reference clock. - stratum 0
NTP server sync time from high performance reference clock - stratum 1
Machine sync time from NTP server - stratum 2

There are 2 category of time sources can be configured
server - one stratum above the local NTP server
peer - At the same stratum level of local NTP server

/etc/chrony.conf file is used to configure chronyd  daemon .

#Sync time from local server
server ntp.example.com iburst

It is recommended to use iburst option , because after service start , four measurement are taken in a short time period for a more accurate initial clock synchronization.

After changing the configuration reset the chronyd

[root@ServerX ]#systemcctl restart chronyd

The chronyc command acts as a client to the chronyd service.
We can use the command chronyc sources -v to verify server is syncing with which NTP server

[root@ServerX ~]# chronyc sources -v
210 Number of sources = 6

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp.example.com              3   6   377    51   +102us[ +242us] +/-  636ms



Tuesday, December 10, 2019

Preserving the systemd journal


By default systemd journal are stored in /run/log/journal, which means it is cleared when the system reboots.

If the directory /var/log/journal exists, the journal will log to that directory instead.

Steps for preserving journal
[root@serverX ~]# mkdir /var/log/journal
[root@serverX ~]# chown root:systemd-journal /var/log/journal
[root@serverX ~]# chmod 2755 /var/log/journal
[root@serverX ~]# killall -USR1 systemd-journald
[root@serverX ~]# ls /var/log/journal/72e8116c885b46de947ad2ca3d0eba76
system.journal  user-1000.journal

journalctl -b  - shows the log of current boot.
journalctl -b -1 - show th log of previous boot

systemd-journald.service and journalctl


systemd-journald.service - It collects information from different sources and loads the messages into the journal.

The systemd journal is not a large text file. It’s a binary file maintained by the daemon. So, it can’t be opened with a text editor. We use journalctl command to view  systemd journal

The systemd journal is stored in /run/log by default and its content are cleared after reboot.

journalctl command shows full system journal, starting with oldest log entty

[root@ServerX~]# journalctl
-- Logs begin at Wed 2019-12-11 04:25:00 AEDT, end at Wed 2019-12-11 07:24:56 AEDT. --
Dec 11 04:25:00 localhost systemd-journal[95]: Runtime journal is using 8.0M (max allowed 756.4M, trying to leave 1.1G free of
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpuset
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpu
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpuacct
Dec 11 04:25:00 localhost kernel: Linux version 3.10.0-693.5.2.el7.x86_64 (mockbuild@x86-041.build.eng.bos.redhat.com) (gcc ve
Dec 11 04:25:00 localhost kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.5.2.el7.x86_64 root=UUID=3ed41454-00c8-480
Dec 11 04:25:00 localhost kernel: e820: BIOS-provided physical RAM map:
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved

Only list log entry of priority err or above

[root@ServerX~]# journalctl -p err
-- Logs begin at Wed 2019-12-11 04:25:00 AEDT, end at Wed 2019-12-11 07:28:30 AEDT. --
Dec 11 04:25:00 localhost kernel: Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Dec 11 04:25:00 localhost kernel: Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Dec 11 04:25:01 localhost iscsid[384]: iSCSI daemon with pid=385 started!
Dec 11 04:25:02 localhost iscsid[384]: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
Dec 11 04:25:02 localhost iscsid[384]: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not c
Dec 11 04:25:02 localhost iscsid[384]: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
Dec 11 04:25:02 localhost iscsid[384]: can't open iscsid.safe_logout configuration file /etc/iscsi/iscsid.conf

In addition to log , there are fields attached to the log entries that can only be seen when verbose output is turned on.
All these filed can be used to filter the output of journal query. some import fields

_COMM=sshd The name of the command
_EXE=/usr/sbin/sshd The path of the executable for the process
_UID=0 UID of the user running the process
_PID=2123 PID of the process
 _SYSTEMD_UNIT=sshd.service  systemd unit that started the process

[root@ServerX~]# journalctl -o verbose
Wed 2019-12-11 04:25:37.940589 AEDT [s=4baf2122c2ee451ea4f5f05ae0dba467;i=566;b=313dc93a97174097905f360bcb417d8e;m=259060a;t=5
    PRIORITY=6
    _UID=0
    _GID=0
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=313dc93a97174097905f360bcb417d8e
    _MACHINE_ID=72e8116c885b46de947ad2ca3d0eba76
    _HOSTNAME=ServerX
    _CAP_EFFECTIVE=1fffffffff
    _TRANSPORT=syslog
    SYSLOG_FACILITY=10
    SYSLOG_IDENTIFIER=sshd
    _COMM=sshd
    _EXE=/usr/sbin/sshd
    _SYSTEMD_CGROUP=/system.slice/sshd.service
    _SYSTEMD_UNIT=sshd.service
    SYSLOG_PID=2123
    MESSAGE=Accepted publickey for ec2-user from 10.0.0.1 port 38186 ssh2: RSA SHA256:GvmOX7imV2RQzRTZ/ojY9jEgf0PeboTkwKx
    _PID=2123
    _CMDLINE=sshd: ec2-user [priv
    _SOURCE_REALTIME_TIMESTAMP=1575998737940589

Some of the important command

journalctl -n - shows last 10 log entries
journalctl -n 5 - shows last 5 log entries

journalctl -p err - Only list log entry of priority err or above
journalctl -u sshd - Show messages for the specified systemd unit UNIT 

journalctl -f - Outputs last 10 lines of journal and continue to output new journal entries like tail -f

journalctl --since today - Shows all log entries that got recorded today
journalctl --since "2012-12-30 20:30:00" --unitl "2010-12-31 12:00:00" - output jounal entry between these 2 dates. Date format YYYY-MM-DD hh:mm:ss. 

jounralctl -o verbose - to show additional field attached to the log entry
journalctl _SYSTEMD_UNIT=sshd.service _PID=2123 - filtering with additional field