Setting up ELK Stack in Linux
Tags: linux, bash, elk
Published: 2023-02-13 05:58:18

## Introduction

Elasticsearch, Logstash, and Kibana are the three open-source tools that make up the ELK Stack.

For the purpose of identifying issues with servers or applications, the ELK stack offers centralized logging. It enables comprehensive log searches in one location.

>Elasticsearch

It was created using Java. An open source search engine. It uses JSON format to get data, search and also save it. It comes in a HTTP dashboard web interface (Kibana).

>Logstash

An open-source tool to manage logs and different events. It saves log data in JSON format then saves it in Elasticsearch.


>Kibana

An open-source tool that let's you visualize the Elaticsearch data in the format of a dashboard.

>Beats

A data transfer service which can be installed on the Client and send a large amount of data from the Client to the Logstash & Elasticsearch server.

## ELK Stack Architecture

A simple ELK Stack looks like


![Simple](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xyv4odbpgocnnrhkvjyl.png)


With Beats it looks like


![Beats](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8kw23p5u2mmnn1a4lwvy.png)

If we are dealing with very large amounts of data, we can add `kafka` and for security we can include `Nginx` as well


![Kafka + Nginx](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/twjpmyrvajeh2sc4dmdt.png)

---

## Setting up the ELK Stack

As it is created from Java, we will need a JVM environment to run ELK. (4GB+ RAM required)

*💡 Selinux can interrupt with retrieving the logs using ELK Stack so we will be disabling that*

```bash
vi /etc/sysconfig/selinux
```

```bash
setenforce 0
```

```bash
reboot 

getenforce
Disabled
```

---

### Step 1 👉 JDK Installation

As Elasticsearch is based on Java, we need the JDK. 

```bash
rpm -qa | grep java
```
We can search for java packages and if we aren't able to find any, we can just install it using `yum`

```bash
yum -y install java-1.8 open-jdk*
```

After the installation, we can check the java version

```bash
java -version
openjdk version "1.8.0_362"
```

---

### Step 2 👉 Elasticsearch Installation

We will first create a directory named `ELK` and work inside there

```bash
mkdir ./ELK

cd ELK
```

Now, we need to download the Elasticsearch package

```bash
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.1-x86_64.rpm
```
*💡 If you don't have `wget` installed, you can install it first using `yum`*

Importing the GPG Key

```bash
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
```
*💡 The GPG Key ensures that the RPM package file was not altered*

We can now install the package that we downloaded

```bash
rpm -ivh elasticsearch-7.6.1-x86_64.rpm
```

```bash
rpm -qa | grep elasticsearch
elasticsearch-7.6.1-1.x86_64
```

We have to edit some configurations by accessing the `elasticsearch` config files and its `yml` file

```bash
vi /etc/elasticsearch/elasticsearch.yml

43 bootstrap.memory_lock: true
55 network.host: localhost
59 http.port: 9200
```

```bash
vi /usr/lib/systemd/system/elasticsearch.service

33 # Memory mlockall
34 LimitMEMLOCK=infinity
```

```bash
vi /etc/sysconfig/elasticsearch
46 MAX_LOCKED_MEMORY=unlimited
```
*💡 As I am setting up the Elasticsearch on the same system as the Logstash, I have set the network host as the `localhost`*

Also enabling the `mlockall` function to change the memory usage setting for all operations to use physical memory

The `mlockall` locks all pages mapped to the address of the call process. It is sometimes used to ensure that the Java virtual machine (JVM) running Elasticsearch does not run out of memory, as this can cause the JVM to crash. 

We will have to reload the daemon to make the `mlockall` setting configured correctly

```bash
systemctl daemon-reload
systemctl enable elasticsearch
systemctl start elasticsearch
```
Enabled and started the `elasticsearch` daemon as well

To confirm JAVA process is running on 9200 port

```bash
netstat -antp | grep 9200
```

Next, we will use `curl` to confirm `mlockall` status

```bash
curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'

{
  "nodes" : {
    "1JrdEndhRaS5Le2JUxiDsw" : {
      "process" : {
       "mlockall" : true
      }
    }
  }
}

```

Also, we can confirm if the `elasticsearch` is running or not

```bash
curl -XGET 'localhost:9200/?pretty'
```

---

### Step 3 👉 Nginx & Kibana Installation

Downloading the `kibana` rpm package

```bash
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.6.1-x86_64.rpm
```

Installing the Kibana package

```bash
rpm -ivh kibana-7.6.1-x86_64.rpm
```

Confirming if the package was successfully installed

```bash
rpm -qa | grep kibana
kibana-7.6.1-1.x86_64
```

We will edit the configuration files for kibana now

```bash
vi /etc/kibana/kibana.yml

2 server.port: 5601
7 server.host: "0.0.0.0"
28 elasticsearch.hosts: ["http://localhost:9200"]
```
*💡 Proceeded with all-network setup to set up the production port and allow external access (only certain hosts can be specified)*

Enabling and starting the `kibana` daemon

```bash
systemctl enable kibana
systemctl start kibana
```

Confirming kibana service status

```bash
netstat -antp | grep 5601
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      8691/node  
```
*💡 It will take some time for the tcp LISTEN node to show*

Now we have to install Nginx 

We will be using the `epel` repository as well so

```bash
yum -y install epel-release
yum -y install nginx httpd-tools
```
The Kibana Service displays information on the screen through the Reverse WEB Proxy Server. That is why we will configure the Reverse WEB Proxy Server with the Nginx Web Server

*💡 A **forward proxy** sits in front of clients and a **reverse proxy** sits in front of servers. Both types of proxies serve as intermediaries between clients and servers*

Editing some entries in the configuration file for Nginx

```bash
vi /etc/nginx/nginx.conf

38 server {
39 listen 80 default_server;
40 listen [::]:80 default_server;
41 server_name _;
42 root /usr/share/nginx/html;
43
44 # Load configuration files for the default server block.
45 include /etc/nginx/default.d/*.conf;
46
47 location / {
48 }
49
50 error_page 404 /404.html;
51 location = /40x.html {
52 }
53
54 error_page 500 502 503 504 /50x.html;
55 location = /50x.html {
56 }
57 }
```
We have to delete the above lines from the Nginx config file as we will be adding a virtual host 

Including the following in kibana configuration file

```bash
server {
 listen 80;
 server_name example.com;
 auth_basic "Restricted Access";
 auth_basic_user_file /etc/nginx/.kibana-user;
 location / {
 proxy_pass http://localhost:5601;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection 'upgrade';
 proxy_set_header Host $host;
 proxy_cache_bypass $http_upgrade;
 }
}
```
This will set up virtual host for Reverse Proxy Server operation

Next, we have to set login credentials for kibana access user

```bash
htpasswd -c /etc/nginx/.kibana-user Admin
New password: 
Re-type new password: 
Adding password for user Admin
```

Verifying Nginx settings

```bash
nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
```

Enabling and starting nginx. We can also confirm if its working

```bash
systemctl enable nginx
systemctl start nginx
netstat -antp | grep nginx
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      8889/nginx: master
```

---

### Step 4 👉 Logstash Installation

Downloading and installing the rpm package

```bash
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.6.1.rpm

rpm -ivh logstash-7.6.1.rpm

rpm -qa | grep logstash
logstash-7.6.1-1.noarch
```

Now, we need to edit the openssl config file

```bash
226 [ v3_ca ]
227 # Server IP Address
228 subjectAltName = IP: <Your-IP-Address>
```
*💡 When sending log information using SSL/TLS, it is recommended to encrypt and transmit log information*

Issuing the openssl certificate

```bash
openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt
```
After generating and verifying the public key certificate and private key used in SSL/TLS, we generated the public key certificate and private key by referring to the SSL/TLS configuration file

```bash
ls -ld /etc/pki/tls/certs/logstash-forwarder.crt
-rw-r--r-- 1 root root 1241  2월 13 13:09 /etc/pki/tls/certs/logstash-forwarder.crt

ls -ld /etc/pki/tls/private/logstash-forwarder.key
-rw-r--r-- 1 root root 1704  2월 13 13:09 /etc/pki/tls/private/logstash-forwarder.key
```

Specifying expiration date and RSA Bit value for each key through additional options

```bash
# Using Filebeat to determine which format to accept data sent from the Client
input {
 beats {
                client_inactivity_timeout => 600
                port => 5044
                ssl => true
                ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
                ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
                  }
        }


# Logstash supports various Filter Plugins (mainly using "grok" Filter Plugins)
 filter {
 if [type] == "syslog" {
 grok {
 match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?: \[%{POSINT:syslog_pid}\])?: 
%{GREEDYDATA:syslog_message}" }
 }
 date {
 match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
 }
}

# Proceeding with settings to transfer data collected by Logstash to Elasticsearch
output {
 elasticsearch {
 hosts => ["localhost:9200"]
 index => "syslog-%{+YYYY.MM.dd}"
 }
}

```

Finally, we just need to enable and start the logstash

```bash
systemctl enable logstash
systemctl start logstash
```

And also, adding 5044 port TCP in the firewall along with the `http` protocol

```bash
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-port=5044/tcp
firewall-cmd --reload
```

We can check the logstash if its working or not

```bash
netstat -antp | grep 5044
tcp6       0      0 :::5044                 :::*                    LISTEN      12868/java  
```

*💡 It will take some time for logstash to start working* 

---

### Step 5 👉 Logstash Client Filebeat Installation

We need the tls certificate inside the Logstash Client, so we will copy the public key certificate from the ELK Server to the Client Linux using `scp`

```bash
# The Client with IP (192.168.1.129)
scp root@192.168.1.128:/etc/pki/tls/certs/logstash-forwarder.crt ./
root@192.168.1.128's password: 
logstash-forwarder.crt                                                                               100% 1241     1.0MB/s   00:00

# Moving the `.crt` file to the `/etc/pki/tls/certs` directory

mv ./logstash-forwarder.crt /etc/pki/tls/certs/
```

Again importing the `GPG` Key for elasticsearch and downloading the filebeat rpm

```bash
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.1-x86_64.rpm

# Installing the filebeat package
rpm -ivh filebeat-7.6.1-x86_64.rpm

rpm -qa | grep filebeat
filebeat-7.6.1-1.x86_64
```
As usual, we have to edit some entries in the config file

```bash
vi /etc/filebeat/filebeat.yml
```

```bash
# Enabling the Filebeat and also declaring what logs we want to see
24 enabled: true
27 paths:
28 - /var/log/*.log
29 - /var/log/secure
30 - /var/log/messages

# Commenting the Elasticsearch output as we will be getting Logstash output
150 #-------------------------- Elasticsearch output ------------------------------
151 #output.elasticsearch:
152 # Array of hosts to connect to.
153 #hosts: ["localhost:9200"]


# Uncommenting the logstash output + declaring the ELK Host IP and the ssl certificate location
163 #----------------------------- Logstash output --------------------------------
164 output.logstash:
165 # The Logstash hosts
166 hosts: ["<ELK HOST IP>:5044"]
167 ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
168 bulk_max_size: 1024 # Specifying the maximum number of events that can be sent at a time
```

We now just have to enable and start the filebeat daemon

```bash
systemctl enable filebeat
systemctl start filebeat
systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-02-13 14:43:53 KST; 2min 49s ago
```

If we check the connection from the ELK Host

```bash
netstat -antp | grep 5044
tcp6       0      0 :::5044                 :::*                    LISTEN      17021/java          
tcp6       0      0 <ELK HOST>:5044      <Client>:35030     ESTABLISHED 17021/java     
```

Now from your browser, you can open the ELK Host server by using its IP address:

![ELK](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/te5abdapsr4nu8l1to2v.png)
Using the username and password that we set before

Once we are in, we will be able to see this screen


![Welcome](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0dvdu99960dc185ax43n.png)

We will click on "Try our sample data" and navigate to


![Index patterns](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i7e3f1cyi9d4y46fwdcb.png)

We just need to Define the index pattern


![Syslog](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hxz7527svkuh0ms45uge.png)

And then configure the settings


![timestamp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xuu2xikxuo76whnh6l5l.png)


Finally, we just need to navigate to the "Discover" tab to see our logs


![Logs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qthghmwudyk6tbvyiwok.png)


✔ We can always filter logs to meet our needs. I demonstrated on how to set up ELK Stack on your Linux Server.