Log Collecting with ELK and Rsyslog
Tags: webdev, tutorial, programming, devlive
Published: 2020-12-14 12:07:30

Recently, we faced the problem of collecting logs from six main and spare servers with a lot of running applications. Every application generates about 8,000 logs per hour. Also, we will soon need to be collecting logs from ~120 Windows machines. At the moment, there are a large number of ready-made solutions that did not meet our requirements. In the end, we chose the simplest and at the same time elegant solution in the form of a collector and a repository of logs — [ELK](https://www.elastic.co/what-is/elk-stack) ([Elasticsearch](https://www.elastic.co/elasticsearch/), [Logstash](https://www.elastic.co/logstash), [Kibana](https://www.elastic.co/kibana)) stack, and as a sender — Rsyslog.

![Image for post](https://dev-to-uploads.s3.amazonaws.com/i/bnycvuv0uixk1o33yfx6.jpeg)

Now let’s take it one after another:

###The problem and the solution:

We have 3 main and 3 spare servers with a lot of launched applications. Every application writes logs in the JSON-strings format into Syslog. In the near future, we will need to collect logs from a lot of Windows machines, but this is a completely different story.

The schematic solution looks like this:

![Log Collecting with ELK and Rsyslog.](https://dev-to-uploads.s3.amazonaws.com/i/2cqm8qx1a47ewtnoua72.png)

Looks good and scalable. Let’s go to preparing and configuring.

###Prepare ELK:

Because we love [Docker](https://www.docker.com/), we will use it! And for our convenience, we will use [docker-compose](https://docs.docker.com/compose/).

Docker-compose file:

    version: '3.2'

    services:
      elasticsearch:
        image: 
    docker.elastic.co/elasticsearch/elasticsearch:7.10.0
        volumes:
          - type: bind
            source: ./elasticsearch/elasticsearch.yml
            target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
    environment:
      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
      discovery.type: single-node
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - elk

      logstash:
        image: docker.elastic.co/logstash/logstash:7.10.0
        volumes:
      - type: bind
        source: ./logstash/config/logstash.yml
        target: /usr/share/logstash/config/logstash.yml
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx512m -Xms512m"
    networks:
      - elk
    depends_on:
      - elasticsearch

      kibana:
        image: docker.elastic.co/kibana/kibana:7.10.0
    volumes:
      - type: bind
        source: ./kibana/kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

    networks:
      elk:
        driver: bridge

    volumes:
      elasticsearch:

We don’t use scaling, because for now, one node is enough. But in the future, we will have to think about scaling.

[Elasticsearch](https://www.elastic.co/elasticsearch/) configs:

    cluster.name: "docker-cluster"
    network.host: 0.0.0.0

    # X-Pack settings
    xpack.license.self_generated.type: basic
    xpack.security.enabled: true
    xpack.monitoring.collection.enabled: true

[Kibana](https://www.elastic.co/kibana/) configs:

    server.name: kibana
    server.host: "0"

    # X-Pack settings
    xpack.monitoring.ui.container.elasticsearch.enabled: true

    # Elasticsearch settings
    elasticsearch.hosts: [ "http://elasticsearch:9200" ]
    elasticsearch.username: elastic
    elasticsearch.password: strongpassword

[Logstash](https://www.elastic.co/logstash/) configs:

    http.host: "0.0.0.0"

    # X-Pack settings
    xpack.monitoring.elasticsearch.hosts: [ 
    "http://elasticsearch:9200" ]
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: elastic
    xpack.monitoring.elasticsearch.password: strongpassword

[Logstash](https://www.elastic.co/logstash/) pipeline configs:

    input {
    udp {
        port => 5000
        codec => "json"
        type => "rsyslog"
    }
    }

    filter {
    json {
        source => "message"
        skip_on_invalid_json => true
    }
    }

    output {
    elasticsearch {
        index => "syslogs-%{+YYYY.MM}"
        hosts => "elasticsearch:9200"
        user => "elasticuser"
        password => "strongpassword"
    }
    }

After all, we can just run our stack.

> docker-compose up -d

After all, containers are started, you can go to the address 127.0.0.1:5601 and if everything is done correctly, we get into the [Kibana](https://www.elastic.co/kibana/) web interface.

###Prepare [Rsyslog](https://www.rsyslog.com/) on servers:

First of all, we need to make sure that all the logs from the applications fall into [Syslog](https://en.wikipedia.org/wiki/Syslog).


Go to */etc/rsyslog.d/* and create two files:

1. File *“01-json-template.conf”*

I will prepare a **template** for outgoing data. Let’s deal with the structure. The first value its timestamp for future filtering. Further, **messages** from the application converted into [JSON](https://en.wikipedia.org/wiki/JSON) format. The next value is **sysloghost**; it’s the name of the server host machine: we need it because, in the future, we will need to separate the logs from different servers. And the last value is **programname**; it’s the name of the application that has written the log.

    template(name="json-template" type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"programname\":\"") property(name="programname")
    constant(value="\"}\n")
    }


2. File *“60-output.conf”*

```if $programname == [“app1”,”app2”,”app3”] then  @elk-server-api:5000;json-template```

> *Array matching working only in rsyslog ≥ 7.2*


###Go to Kibana

First, to configure Kibana, go into Management/Index Patterns and create Pattern ``syslogs-*``. After this, Kibana will find all our log indexes.

In the Kibana Discover page, we can use [Kibana Query Language(KQL)](https://www.elastic.co/guide/en/kibana/current/kuery-query.html) for selecting and filtering logs. Examples:

* Get logs only from “Server2”:

> *sysloghost : “Server1”*

* Get “App2” logs from “Server1”

> *sysloghost : “Server1” and programname : “App2”*

* Filter by ``level`` :

> *sysloghost : “Server1” and programname : “App2” and level : “info”*



P.S. Don’t forget to configure your [Index Lifecycle Policies (ILM)](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/index-lifecycle-management.html).


Our example of the [Index Lifecycle Policies](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/index-lifecycle-management.html):

![The Example of the Index Lifecycle Policies.](https://dev-to-uploads.s3.amazonaws.com/i/lzq6j42147ijppufeinb.png)

![The Example of the Index Lifecycle Policies.](https://dev-to-uploads.s3.amazonaws.com/i/xw49kgzektr6bkrr4vom.png)

That’s all. For more convenient log monitoring, we have set up the simple [Kibana Dashboard](https://www.elastic.co/guide/en/kibana/current/dashboard.html). It looks like this:

![Kibana Dashboard.](https://dev-to-uploads.s3.amazonaws.com/i/zztyfywpb4ac2dprle1l.jpeg)

##Conclusion

To be collecting logs from different servers and services, we can use a simple solution with [rsyslog](https://www.rsyslog.com/) and [ELK](https://www.elastic.co/what-is/elk-stack). As a result, we get a fast, flexible, and scalable system to monitor and find the logs we need.

[![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/35chc3ikocyuc6z5ttrk.jpeg)](https://maddevs.io/?utm_source=blog&utm_medium=log-collecting-elk-rsyslog)

Previously published at [maddevs.io.](https://maddevs.io/blog/log-collecting-with-elk-and-rsyslog?utm_source=devto&utm_medium=log-collecting-elk-rsyslog)