Monitor Linux Servers Using Prometheus Node Exporter

 Setup Node Exporter Binary

1. Download the latest node exporter package. You should check the Prometheus downloads section for the latest version and update this command to get that package.

2. Download the node_exporter using wget and untar the file

[root@server1 ~]# wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
[root@server1 ~]# tar -xvzf node_exporter-1.3.1.linux-amd64.tar.gz 

3. Move the node export binary to /usr/local/bin

[root@server1 ~]# mv node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin/

Create a Custom Node Exporter Service

1. Create a node_exporter user to run the node exporter service.

[root@server1 ~]# useradd -rs /bin/false node_exporter

2. Create a node_exporter service file under systemd.

[root@server1 ~]# vi /etc/systemd/system/node_exporter.service 

3. Add the following service file content to the service file and save it.

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target 

4.  Reload the system daemon and start the node exporter service.

[root@server1 ~]# systemctl daemon-reload
[root@server1 ~]# systemctl start node_exporter

5. Enable the node_exporter service and check the status.

[root@server1 ~]# systemctl enable node_exporter
[root@server1 ~]# systemctl status node_exporter

6. Enable the port 9100/tcp in firewall.

[root@server1 ~]# firewall-cmd --permanent --add-port=9100/tcp
success
[root@server1 ~]# firewall-cmd --reload
success 

7. You can see all the server metrics by visiting your server URL on /metrics as shown below.

http://<server-IP>:9100/metrics

Configure the Server as Target on Prometheus Server

Now that we have the node exporter up and running on the server, we have to add this server as target on the Prometheus server configuration.

Note: This configuration should be done on the Prometheus server.

1. Login to the Prometheus server and open the prometheus.yml file.

[root@promotheus-server ~]# vi /etc/prometheus/prometheus.yml

2. Under the scrape config section add the node exporter target as shown below. Change 192.168.31.252 with your server IP where you have setup node exporter. Job name can be your server hostname or IP for identification purposes.

- job_name: 'node_exporter_metrics'
  scrape_interval: 5s
  static_configs:
    - targets: ['192.168.31.252:9100']

3. Restart the prometheus service for the configuration changes to take place.

[root@promotheus-server ~]# systemctl restart prometheus

4. Now, if you check the target in prometheus web UI (http://<prometheus-IP>:9090/targets) , you will be able to see the status as shown below.

5. Also, you can use the Prometheus expression browser to query for node related metrics. Following are the few key node metrics you can use to find its statistics.

node_memory_MemFree_bytes
node_cpu_seconds_total
node_filesystem_avail_bytes
rate(node_cpu_seconds_total{mode="system"}[1m]) 
rate(node_network_receive_bytes_total[1m])

Prometheus Setup on Linux Server

What is  Prometheus ?

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database  built using a HTTP pull model, with flexible queries and real-time alerting.

Installing and configuring Prometheus?

1. Go to the official Prometheus downloads page and get the latest download link for the Linux binary.

2. Download the source using wget, untar it, and rename the extracted folder to prometheus-files.

[root@server1 ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.32.1/prometheus-2.32.1.linux-amd64.tar.gz
[root@server1 ~]# tar -xvzf prometheus-2.32.1.linux-amd64.tar.gz
[root@server1 ~]# mv prometheus-2.32.1.linux-amd64 prometheus-files

3. Create a Prometheus user, required directories, and make Prometheus the user as the owner of those directories.

[root@server1 ~]# useradd --no-create-home --shell /bin/false prometheus
[root@server1 ~]# mkdir /etc/prometheus
[root@server1 ~]# mkdir /var/lib/prometheus
[root@server1 ~]# chown prometheus:prometheus /etc/prometheus
[root@server1 ~]# chown prometheus:prometheus /var/lib/prometheus

4. Copy prometheus and promtool binary from prometheus-files folder to /usr/local/bin and change the ownership to prometheus user.

[root@server1 ~]# cp prometheus-files/prometheus /usr/local/bin/
[root@server1 ~]# cp prometheus-files/promtool /usr/local/bin/
[root@server1 ~]# chown prometheus:prometheus /usr/local/bin/prometheus
[root@server1 ~]# chown prometheus:prometheus /usr/local/bin/promtool

5. Move the consoles and console_libraries directories from prometheus-files to /etc/prometheus folder and change the ownership to prometheus user.

[root@server1 ~]# cp -r prometheus-files/consoles /etc/prometheus
[root@server1 ~]# cp -r prometheus-files/console_libraries /etc/prometheus
[root@server1 ~]# chown -R prometheus:prometheus /etc/prometheus/consoles
[root@server1 ~]# chown -R prometheus:prometheus /etc/prometheus/console_libraries

Setup Prometheus Configuration

All the prometheus configurations should be present in /etc/prometheus/prometheus.yml file.

1. Create the prometheus.yml file.

[root@server1 ~]# vi /etc/prometheus/prometheus.yml

2. Copy the following contents to the prometheus.yml file.

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

3. Change the ownership of the file to prometheus user.

[root@server1 ~]# chown prometheus:prometheus /etc/prometheus/prometheus.yml

Setup Prometheus Service File

1. Create a prometheus service file.

[root@server1 ~]# vi /etc/systemd/system/prometheus.service

2. Copy the following content to the file.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

3. Reload the systemd service to register the prometheus service and start the prometheus service.

[root@server1 ~]#  systemctl daemon-reload
[root@server1 ~]#  systemctl start prometheus

4. Check the prometheus service status using the following command.

[root@server1 ~]# systemctl status prometheus.service
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-01-19 05:02:15 EST; 35min ago
 Main PID: 1220 (prometheus)
    Tasks: 8 (limit: 4304)
   Memory: 124.4M
   CGroup: /system.slice/prometheus.service
           └─1220 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml

5. Open Firewall port 9090/tcp.

[root@server1 ~]# firewall-cmd --permanent --add-port=9090/tcp
success
[root@server1 ~]# firewall-cmd --reload
success 

Access Prometheus Web UI

Now you will be able to access the prometheus UI on 9090 port of the prometheus server.

http://<prometheus-ServerIp>:9090/graph 

Right now, we have just configured the Prometheus server. You need to register the target in the prometheus.yml file to get the metrics from the source systems.

For example, if you want to monitor ten servers, the IP address of these servers should be added as a target in the Prometheus configuration to scrape the metrics.

The server should have Node Exporter installed to collect all the system metrics and make it available for Prometheus to scrap it.