Tuesday, December 10, 2019

Preserving the systemd journal


By default systemd journal are stored in /run/log/journal, which means it is cleared when the system reboots.

If the directory /var/log/journal exists, the journal will log to that directory instead.

Steps for preserving journal
[root@serverX ~]# mkdir /var/log/journal
[root@serverX ~]# chown root:systemd-journal /var/log/journal
[root@serverX ~]# chmod 2755 /var/log/journal
[root@serverX ~]# killall -USR1 systemd-journald
[root@serverX ~]# ls /var/log/journal/72e8116c885b46de947ad2ca3d0eba76
system.journal  user-1000.journal

journalctl -b  - shows the log of current boot.
journalctl -b -1 - show th log of previous boot

systemd-journald.service and journalctl


systemd-journald.service - It collects information from different sources and loads the messages into the journal.

The systemd journal is not a large text file. It’s a binary file maintained by the daemon. So, it can’t be opened with a text editor. We use journalctl command to view  systemd journal

The systemd journal is stored in /run/log by default and its content are cleared after reboot.

journalctl command shows full system journal, starting with oldest log entty

[root@ServerX~]# journalctl
-- Logs begin at Wed 2019-12-11 04:25:00 AEDT, end at Wed 2019-12-11 07:24:56 AEDT. --
Dec 11 04:25:00 localhost systemd-journal[95]: Runtime journal is using 8.0M (max allowed 756.4M, trying to leave 1.1G free of
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpuset
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpu
Dec 11 04:25:00 localhost kernel: Initializing cgroup subsys cpuacct
Dec 11 04:25:00 localhost kernel: Linux version 3.10.0-693.5.2.el7.x86_64 (mockbuild@x86-041.build.eng.bos.redhat.com) (gcc ve
Dec 11 04:25:00 localhost kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.5.2.el7.x86_64 root=UUID=3ed41454-00c8-480
Dec 11 04:25:00 localhost kernel: e820: BIOS-provided physical RAM map:
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
Dec 11 04:25:00 localhost kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved

Only list log entry of priority err or above

[root@ServerX~]# journalctl -p err
-- Logs begin at Wed 2019-12-11 04:25:00 AEDT, end at Wed 2019-12-11 07:28:30 AEDT. --
Dec 11 04:25:00 localhost kernel: Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Dec 11 04:25:00 localhost kernel: Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Dec 11 04:25:01 localhost iscsid[384]: iSCSI daemon with pid=385 started!
Dec 11 04:25:02 localhost iscsid[384]: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
Dec 11 04:25:02 localhost iscsid[384]: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not c
Dec 11 04:25:02 localhost iscsid[384]: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
Dec 11 04:25:02 localhost iscsid[384]: can't open iscsid.safe_logout configuration file /etc/iscsi/iscsid.conf

In addition to log , there are fields attached to the log entries that can only be seen when verbose output is turned on.
All these filed can be used to filter the output of journal query. some import fields

_COMM=sshd The name of the command
_EXE=/usr/sbin/sshd The path of the executable for the process
_UID=0 UID of the user running the process
_PID=2123 PID of the process
 _SYSTEMD_UNIT=sshd.service  systemd unit that started the process

[root@ServerX~]# journalctl -o verbose
Wed 2019-12-11 04:25:37.940589 AEDT [s=4baf2122c2ee451ea4f5f05ae0dba467;i=566;b=313dc93a97174097905f360bcb417d8e;m=259060a;t=5
    PRIORITY=6
    _UID=0
    _GID=0
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=313dc93a97174097905f360bcb417d8e
    _MACHINE_ID=72e8116c885b46de947ad2ca3d0eba76
    _HOSTNAME=ServerX
    _CAP_EFFECTIVE=1fffffffff
    _TRANSPORT=syslog
    SYSLOG_FACILITY=10
    SYSLOG_IDENTIFIER=sshd
    _COMM=sshd
    _EXE=/usr/sbin/sshd
    _SYSTEMD_CGROUP=/system.slice/sshd.service
    _SYSTEMD_UNIT=sshd.service
    SYSLOG_PID=2123
    MESSAGE=Accepted publickey for ec2-user from 10.0.0.1 port 38186 ssh2: RSA SHA256:GvmOX7imV2RQzRTZ/ojY9jEgf0PeboTkwKx
    _PID=2123
    _CMDLINE=sshd: ec2-user [priv
    _SOURCE_REALTIME_TIMESTAMP=1575998737940589

Some of the important command

journalctl -n - shows last 10 log entries
journalctl -n 5 - shows last 5 log entries

journalctl -p err - Only list log entry of priority err or above
journalctl -u sshd - Show messages for the specified systemd unit UNIT 

journalctl -f - Outputs last 10 lines of journal and continue to output new journal entries like tail -f

journalctl --since today - Shows all log entries that got recorded today
journalctl --since "2012-12-30 20:30:00" --unitl "2010-12-31 12:00:00" - output jounal entry between these 2 dates. Date format YYYY-MM-DD hh:mm:ss. 

jounralctl -o verbose - to show additional field attached to the log entry
journalctl _SYSTEMD_UNIT=sshd.service _PID=2123 - filtering with additional field

Friday, September 8, 2017

Network Configuration in RHEL 7 or CentOS 7

Network Interface names


Traditional network interfacin Linux are named as eth0,eth1,eth2..
This is changed in RHEL 7 . RHEL 7 assign fixed names based on firmware, device topology and device type.

Initial 2 character

Ether interface begins with en
WLAN interface begins with wl
WWAN interface begins with ww

The next character(s) represent the type of adapter

On-board - o
Hotplug slot - s
PCI Geographical locations - p
Incorporate a MAC address - x

Finally, a Number N is used to represent an index, ID , or port.

Examples
eno1 - First embedded on-board network interface 
enp2s0 - PCI card network interfcae 

If the fixed name cannot determined then the tradition name such as ethN will be used.

Network Manager

                  In RHEL7 , Configuration of network interface is managed by a system daemon called Network Manager. 

                  For Network Manager
                  device - A device is network interface
                  connection - A connection is a collection of setting that can be configured for a device

  • Only one connection can be active for any one device at a time.
  • The persistent configuration are stored in /etc/sysconfig/netowork-script/ifcfg-name
  • nmcli command can be used to create and edit connection files from the shell prompt

Viewing network information 

The command nmcli dev status will show the status of all network devices.


[root@rhel7-server1 ~]# nmcli dev status
DEVICE      TYPE      STATE      CONNECTION
virbr0      bridge    connected  virbr0
enp0s3      ethernet  connected  enp0s3
lo          loopback  unmanaged  --
virbr0-nic  tun       unmanaged  --

The command nmcli con show will show a list of all connections.
 
[root@rhel7-server1 ~]# nmcli connection show
NAME     UUID                                  TYPE            DEVICE
enp0s3   b66164cd-6dac-4f5c-af32-f742fe090608  802-3-ethernet  enp0s3
virbr0   a35a0cdf-b10c-41e6-aec3-ec8de78e1b79  bridge          virbr0
default  48b8ebda-6e50-4b20-affe-a7afc91e8136  802-3-ethernet  --


The command nmcli con show --active will show a list of active connections.

[root@rhel7-server1 ~]# nmcli connection show --active
NAME    UUID                                  TYPE            DEVICE
enp0s3  b66164cd-6dac-4f5c-af32-f742fe090608  802-3-ethernet  enp0s3
virbr0  a35a0cdf-b10c-41e6-aec3-ec8de78e1b79  bridge          virbr0


The see the details of the connection specify the connection ID (name)


[root@rhel7-server1 ~]# nmcli  connection show "enp0s3"
...
ipv4.method:                            auto
ipv4.dns:
ipv4.dns-search:
ipv4.dns-options:                       (default)
ipv4.dns-priority:                      0
ipv4.addresses:
ipv4.gateway:                           --
ipv4.routes:
ipv4.route-metric:                      -1
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --

Command to show device details
[root@rhel7-server1 ~]# nmcli dev show enp0s3
GENERAL.DEVICE:                         enp0s3
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         08:00:27:78:58:3D
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     enp0s3
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/0
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.18.10/24
IP4.GATEWAY:                            192.168.18.1
IP4.DNS[1]:                             192.168.18.1
IP6.ADDRESS[1]:                         fe80::a00:27ff:fe78:583d/64
IP6.GATEWAY:

Creating Network connections with nmcli


When creating new connection with nmcli , the order of the argument is important..
The command argument comes first and must include type and interface.
Next specify type specific argument and finally specify IP address,prefix and gateway.
Addition setting such as DNS Server are set as modifications once the connection exists.

Define a new connection name "default" which will auto connect as an Ethernet connection on the enp0s3 device using DHCP.

[root@rhel7-server1 ~]# nmcli con add con-name "default" type ethernet ifname enp0s3
Connection 'default' (48b8ebda-6e50-4b20-affe-a7afc91e8136) successfully added.

Create new network connection name "static" and specify the IP address and gateway. Do not autoconnect
[root@rhel7-server1 ~]# nmcli con add con-name "static" type ethernet ifname enp0s3 \
autoconnect no ip4 192.168.18.100 gw4 192.168.18.1
Connection 'static' (547ba59d-6378-492b-9456-617c44f94136) successfully added.

Use the below command to change to "static" connection.

[root@rhel7-server1 ~]# nmcli connection up "static"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)
[root@rhel7-server1 ~]# nmcli dev status
DEVICE      TYPE      STATE      CONNECTION
virbr0      bridge    connected  virbr0
enp0s3      ethernet  connected  static
lo          loopback  unmanaged  --
virbr0-nic  tun       unmanaged  --


Below command to change it back to DHCP connection

[root@rhel7-server1 ~]# nmcli connection up "default"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/8)
[root@rhel7-server1 ~]# nmcli device status
DEVICE      TYPE      STATE      CONNECTION
virbr0      bridge    connected  virbr0
enp0s3      ethernet  connected  default
lo          loopback  unmanaged  --
virbr0-nic  tun       unmanaged  --


We can get some help using the below command


[root@rhel7-server1 ~]# nmcli connection add help
Usage: nmcli connection add { ARGUMENTS | help }

ARGUMENTS := COMMON_OPTIONS TYPE_SPECIFIC_OPTIONS SLAVE_OPTIONS IP_OPTIONS [-- ([+|-]. )+]

  COMMON_OPTIONS:
                  type 
                  ifname  | "*"
                  [con-name ]
                  [autoconnect yes|no]
                  [save yes|no]
                  [master ]
                  [slave-type ]

  TYPE_SPECIFIC_OPTIONS:
    ethernet:     [mac ]
                  [cloned-mac ]
                  [mtu ]



Modifying Network connections with nmcli


nmcli con mod argument used to modify connection setting

Comparison of nm-setting and ifcfg-* directives.
nmcli con mod
Ifcfg-* file
Effect
ipv4.method manual
BOOTPROTO=none
IPv4 address configured statically
ipv4.method auto
BOOTPROTO=dhcp
Will look for configuration from DHCPv4 server.If static address also set , will not bring those up until we have information from DHCPv4
Ipv4.addresses “192.0.2.1/24 192.0.2.254”
IPADDR0=192.0.2.1
PREFIX0=24
GATEWAY0=192.0.2.254
Sts static IPv4 address,Network Prefix and default gateway.
Ipv4.dns 8.8.8.8
DNS0=8.8.8.8
Modify /etc/resolv.conf to use this nameserver
Ipv4.dns-search example.com
DOMAIN=example.com
Modify /etc/resolv.conf to use this domain in the search directive.
Ipv4.ignore-auto-dns true
PEERDNS=no
Ignore DNS server information from DHCP server
Connection.autoconnect yes
ONBOOT=yes
Automatically activates this connection at boot
Connection.id eth0
NAME=eth0
The name of this connection is eth0
Connection.interface-name eth0
DEVICE=eth0
The connection is bound to the network interface with this name
802-3-ethernet.mac-address …
HWADDR=…
The connection is bound to the network interface with this MAC Address


Examples

Turn off autoconnect
[root@rhel7-server1 ~]#  nmcli connection mod "static" connection.autoconnect no

Specify a DNS server
[root@rhel7-server1 ~]# nmcli con mod "static" ipv4.dns 192.168.18.1 
[root@rhel7-server1 ~]# nmcli connection show static | grep -w ipv4.dns:
ipv4.dns:                               192.168.18.1

Adding a secondary DNS.
We can add or remove by using "+" or "-' in front of the argument.
[root@rhel7-server1 ~]# nmcli con mod "static" +ipv4.dns 8.8.8.8
[root@rhel7-server1 ~]# nmcli connection show static | grep -w ipv4.dns:
ipv4.dns:                               192.168.18.1,8.8.8.8

To modify the IP address and gateway
[root@rhel7-server1 ~]# nmcli con mod "static" ipv4.addresses "192.168.18.101/24" \ 
ipv4.gateway "192.168.18.1"


Adding secondary IP address without gateway
[root@rhel7-server1 ~]# nmcli con mod static +ipv4.addresses 192.168.18.100/24
[root@rhel7-server1 ~]# nmcli con show "static" | grep -w ipv4.addresses:
ipv4.addresses:                         192.168.18.101/24, 192.168.18.100/24


Summary

Command  Use
nmcli dev status List all devices
nmcli con show List all connections
nmcli con up "ID" Activate a connection
nmcli con down "ID"  Deactivate a connection .
The connection will restart if autoconnect is yes
nmcli dev dis  DEV Bring down an interface and temp disable autoconnect.
nmcli net off Disable all managed interfaces
nmcli con add Add a new connection
nmcli con mod "ID" … Modify a connection
nmcli con del "ID" Delete a connection



Editing Network configuration files


Network configuration files are place in /etc/sysconfig/network-scripts
[root@rhel7-server1 ~]# ll /etc/sysconfig/network-scripts/ifcfg-*
-rw-r--r--. 1 root root 312 Dec 14 00:12 /etc/sysconfig/network-scripts/ifcfg-default
-rw-r--r--. 1 root root 310 Dec 14 00:12 /etc/sysconfig/network-scripts/ifcfg-enp0s3
-rw-r--r--. 1 root root 254 Sep 12  2016 /etc/sysconfig/network-scripts/ifcfg-lo
-rw-r--r--. 1 root root 437 Dec 14 00:24 /etc/sysconfig/network-scripts/ifcfg-static

Edit the network configuration files
[root@rhel7-server1 ~]# echo "IPADDR2=192.168.18.200" >> /etc/sysconfig/network-scripts/ifcfg-static
[root@rhel7-server1 ~]# echo "PREFIX2=24" >> /etc/sysconfig/network-scripts/ifcfg-static

Reload the configuration chagnes
[root@rhel7-server1 ~]# nmcli con reload

Restart the connection with new setting
[root@rhel7-server1 ~]# nmcli con up "static"

Check the ip address details
[root@rhel7-server1 ~]# ip addr show enp0s3
2: enp0s3:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:78:58:3d brd ff:ff:ff:ff:ff:ff
    inet 192.168.18.100/24 brd 192.168.18.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 192.168.18.101/24 brd 192.168.18.255 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet 192.168.18.200/24 brd 192.168.18.255 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::455e:6fdd:ed5f:eff4/64 scope link
       valid_lft forever preferred_lft forever


Thursday, September 7, 2017

Reset Root Password in RHEL 7 and CentOS 7


1. Restart the machine using Ctrl + Alt + Del and Interrupt the count down by pressing any key
2. Select the default menu and press “e” to edit the current grub entry.
3. Using the cursor navigate to line “linux16” and add entry “rd.break” at the end. This will break just before control is handed from the intramfs to the actual system.
4. Press “Ctrl+x” to boot using the modified config
5. At the switch_root prompt, remount the /sysroot file system in read-write mode.
    #mount –o remount,rw /sysroot 
6. Using the command “chroot /sysroot” to change the current root directory
7. Change the password using “passwd” command
8. Configure the system to automatically perform a full SElinux relabel after reboot. Since the password command recreated the /etc/shadow file without any SElinux contest.
    #touch /.autorelabel
9. Type “exit” twice and reboot the system in normal mode.

Thursday, February 19, 2015

df and du showing different Size

df and du shows different Size and recovering the space without killing the process

1. From lsof we can find the big 10 files which is deleted. And the first file is around  ((2407845888 / 1024 ) /1024) = 2296 MB
       
[root@server1  fd]# lsof | grep deleted | sort -nr -k 7 | head
frmweb    21199    mislve   23u      REG      253,0 2407845888     819375 /tmp/filesvuEFz.TMP (deleted)
frmweb    28933    mislve   27u      REG      253,0    4032864     819378 /tmp/fileiTj11l.TMP (deleted)
frmweb    16365    mislve   24u      REG      253,0    2248832     819483 /tmp/file5z0oen.TMP (deleted)
frmweb    28697    mislve   27u      REG      253,0     959488     819238 /tmp/fileemrhVz.TMP (deleted)
frmweb    24794    mislve   27u      REG      253,0     906304     819492 /tmp/file4d68i0.TMP (deleted)
frmweb    25383    mislve   24u      REG      253,0     393216     819572 /tmp/file7powEH.TMP (deleted)
frmweb    25329    mislve   28u      REG      253,0     386784     819355 /tmp/filembs5IG.TMP (deleted)
frmweb    24400    mislve   28u      REG      253,0     307328     819228 /tmp/fileL8p1m7.TMP (deleted)
frmweb     2717    mislve   27u      REG      253,0     277472     819290 /tmp/fileVzblkn.TMP (deleted)
frmweb     2002    mislve   24u      REG      253,0     262784     819516 /tmp/fileUeZpzA.TMP (deleted)


2. One option is to kill the process or nullfy the File Descripter (FD).We can find the FD as shown below.

       
[root@server1 fd]# ls -l /proc/21199/fd/ | grep deleted
lrwx------ 1 mislve dba 64 Feb 19 13:00 23 -> /tmp/filesvuEFz.TMP (deleted)
       

3.Nullify the file descripter

[root@server1 fd]#  echo /dev/null >  /proc/21199/fd/23
 

4. Size of the Fileystem before and after
     

Before
[root@server1 ~]# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       11G  5.8G  4.3G  58% /

After
[root@server1 ~]# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       11G  3.6G  6.5G  36% /

Sunday, January 11, 2015

Enabling Virtualization option in Lenovo G50

Enabling Virtualization option in BIOS 
1. I bought Lenovo G50 AMD A8 / 8GB RAM / 1TB HDD and my plan is to install some Hypervisor virtually using VMWare workstation and run some VM in that, but I was not able to see the option in BIOS to enable Virtualization
3.You can enable  Virtualization option by updating BIOS using this link.
4.After installing this BIOS update you can see option to enable Virtualization option.
Installing VMWare vSphere client in Windows 8.1
Next issue I was facing is installing vSphere Client in Windows 8.1. It keeps crashing.
The installation of file,VMware-viclient.exe, keeps crashing during installation on Windows 8.1.  This was caused by not having the .NET Framework x3.5 installed.
The cause of issue is that the .NET framework v3.5 is not installed in Windows 8 by default, and is a required component for the vSphere client.  I assume that the redistributable is packaged with the vSphere client installer, but it doesn’t work in Windows 8.1.
“The .NET Framework v3.5 is easy to install, just issue the following command from an elevated/administrator command prompt:
Dism /online /enable-feature /featurename:NetFx3 /All /Source:F:\sources\sxs /LimitAccess
(replace F:\ with the DVD drive where your Windows 8.1 is located.  If your DVD drive is as slow as mine, just copy the ISO to your hard drive, mount it with daemon-tools, and use that location instead.)
After that, the vSphere client happily installed and I’m off to play with Windows 8.1 now.”
Issue in Installing VM in ESXi server
After Installing vSphere client I was able to connect to ESXi Server which was installed in VMWare workstation 10, but I was not able to install VM in that.
When I created VM and started it gives the error "VMware ESX unrecoverable error: (vcpu-0) vcpu-0:NOT_IMPLEMENTED vmcore/vmm/cpu/fpu.c:452"
To solve this issue you need to add the following line in /etc/vmware/config in ESXi Server

featMask.mockup.cpuid.NUMLEVELS = Val:0xb
featMask.mockup.cpuid.AVX = Val:0
featMask.mockup.cpuid.XSAVE = Val:0

Note: This option disable AVX (Advanced Vector Extensions)

Reference :
https://communities.vmware.com/thread/476180?start=0&tstart=0
http://www.brianwilliams.pro/installing-vsphere-client-on-windows-8-1/

Friday, May 23, 2014

Vyatta Firewall Configuration backup

Vyatta Firewall Automated Backup 


1. Script to backup the vyatta configuration thorugh ssh. Save this file as /home/vyatta/backup_expect.sh.
Modify User name ,Password and IP address details.

#!/usr/bin/expect 
set timeout 1
spawn $env(SHELL)
send "configure\r"
expect -re  ".*# $"
sleep 5
send "save scp://vyatta:password@[ip-address]/home/vyatta/R1-config.boot.`date +%Y%m%d-%H%M`\r"
expect -re  ".*# $"
send "exit\r"
exit
expect eof

2.Enable the crontab entry for vyatta user

SHELL=/bin/vbash
0 20 * * * /home/vyatta/backup_expect.sh > /tmp/backup-out