How to setup local development with ELK observability tools using docker-compose
Tags: devops, development, kibana, programming
Published: 2024-02-05 12:26:17

A lot of observability is done using cloud provider tools that are one subscription away, yet replicating them for local development can be challenging. At Woovi, we prioritize aligning our development environment with production. That's why we run locally all services available in production using Docker Compose. Observability is not an exception. In this article, we'll delve into the detailed setup of ELK observability tools in our local development environment.

## ELK Tools

This are the tools we use for observability in our stack.

- APM
  Application performance monitoring.

- Elastic Search
  Search Engine.

- Kibana
  Visualization Platform

- Logstash
  Log ingestion

- MetricBeat
  Server data shipper

- Filebeat
  Log harvester and aggregator

- Fleet
  Agents integration

Elastic Integrations offers an unified way to ingrate features of ELK stack, managed by fleet servers you can integrate over Kibana all integrations available using elastic agents, and manage them using policies trough an unified interface. In order to use this features we need security enable in kibana and elastic search and we need to provide certificate authentication for each service. In order to simplify we will be using the same certificate for all services, but this is only suited for local development and should not be used in production.

First we will need configuration files for some services, the file structure will be:

```bash
├── .env

├── docker-compose.yml

├──|observability |──  filebeat.yml

├── |observability |── logstash.conf

└── |observability |── metricbeat.yml

```

## Docker Compose

The docker compose will be responsible for setting up certificates and copying to the volume of each service, that way we can setup SSL between services.

```yml
version: "3.8"
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: fleet-server\n"\
          "    dns:\n"\
          "      - fleet-server\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
      - ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_CERTIFICATE=config/certs/kibana/kibana.crt
      - SERVER_SSL_KEY=config/certs/kibana/kibana.key
      - SERVER_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}
    mem_limit: ${KB_MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -I -s --cacert config/certs/ca/ca.crt https://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  metricbeat01:
    depends_on:
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/beats/metricbeat:${STACK_VERSION}
    user: root
    volumes:
      - certs:/usr/share/metricbeat/certs
      - metricbeatdata01:/usr/share/metricbeat/data
      - "./observability/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
      - KIBANA_HOSTS=https://kibana:5601
      - LOGSTASH_HOSTS=http://logstash01:9600
      - CA_CERT=certs/ca/ca.crt
      - ES_CERT=certs/es01/es01.crt
      - ES_KEY=certs/es01/es01.key
      - KB_CERT=certs/kibana/kibana.crt
      - KB_KEY=certs/kibana/kibana.key
    command: -strict.perms=false

  filebeat01:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/beats/filebeat:${STACK_VERSION}
    user: root
    volumes:
      - certs:/usr/share/filebeat/certs
      - filebeatdata01:/usr/share/filebeat/data
      - "./filebeat_ingest_data/:/usr/share/filebeat/ingest_data/"
      - "./observability/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
      - KIBANA_HOSTS=https://kibana:5601
      - LOGSTASH_HOSTS=http://logstash01:9600
      - CA_CERT=certs/ca/ca.crt
    command: -strict.perms=false

  logstash01:
    depends_on:
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - certs:/usr/share/logstash/certs
      - logstashdata01:/usr/share/logstash/data
      - "./logstash_ingest_data/:/usr/share/logstash/ingest_data/"
      - "./observability/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro"
    environment:
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200

  fleet-server:
    depends_on:
      kibana:
        condition: service_healthy
      es01:
        condition: service_healthy
    image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
    volumes:
      - certs:/certs
      - fleetserverdata:/usr/share/elastic-agent
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    ports:
      - ${FLEET_PORT}:8220
      - ${APMSERVER_PORT}:8200
    user: root
    environment:
      - SSL_CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - FLEET_CA=/certs/ca/ca.crt
      - FLEET_ENROLL=1
      - FLEET_INSECURE=true
      - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
      - FLEET_SERVER_ELASTICSEARCH_INSECURE=true
      - FLEET_SERVER_ENABLE=1
      - FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
      - FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
      - FLEET_SERVER_INSECURE_HTTP=true
      - FLEET_SERVER_POLICY_ID=fleet-server-policy
      - FLEET_URL=https://fleet-server:8220
      - KIBANA_FLEET_CA=/certs/ca/ca.crt
      - KIBANA_FLEET_SETUP=1
      - KIBANA_FLEET_USERNAME=elastic
      - KIBANA_FLEET_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_HOST=https://kibana:5601

volumes:
  elastic_data: {}
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
  metricbeatdata01:
    driver: local
  filebeatdata01:
    driver: local
  logstashdata01:
    driver: local
  fleetserverdata:
    driver: local
  cache:
    driver: local

networks:
  default:
    name: elastic
    external: false
```

Its not as complicated as it looks, some of the things the compose file does by service are:

- Setup

Create an instance of an Elastic Search service to use on validation of the certificates. Then, via bash script we create the certificates using `elasticsearch-certutil`, here we specify what are our cluster nodes and servers instances creating CA cert and node certs giving the correct file permissions and distributing the cert and keys. We test the authentication worked using elastic search service and then the service stops.

- ES01

  Main Elastic Search service node, xpack.security options need to be enabled and pointing to the correct folder with the node certificate and certificate authority. A basic health check is passed using auth validation as main test parameter.

- Kibana

  Kibana service, security config is the same as ES, xpack need to be enabled, extra configuration will be passed in its own yml file. To Access this file we pass it as bind read only volume.
  `./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro`
  We add internal apm traces so once the initial setup go up we can verify kibana for itself.

- filebeat01

  For filebeat we bind the volumes `/var/lib/docker/containers` and `/var/run/docker.sock` allowing filebeat to get all containers internal logs, the config in `observability/filebeat.yml` will set autodiscover for that purpose, using docker as a provider, will also config kibana and ES. Ingest data input and ES outupt.

- metricbeat01

  For Metricbeat the binding mounts are to allow getting host system metrics and information and sending to elastic search.

- logstash01

  For logstash the bind mount config file will setup ingest plugins and output to elasti search. We are using basic starter configuration but in the config file is where you will setup formating. The ingest data path you can send the log file to be read using the strategy described in the config file.

- fleet-server

  Fleet server will manage agents and its integrations.

### Running Stack

1 - Run:


```
docker-compose up -d
```

Wait untill is finished:

![Output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lyi5xf470seqrok8obiy.png)

2 - Open [kibana](https://localhost:5601/app/home#/)

Enter username and password setup in .env file

3 - Go to Fleet/Management

![Fleet](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/km1j0h7rn0apqsr9nqxh.png)

4 - Go to Settings


![Settings](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/obh2i4rrnvvyuuhemk6f.png)



5 - Edit the output agent


![Edit Output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ymnbu4gl8riewcjdbfp5.png)



You will need 3 items to configure the certificate.

Url: `https://es01:9200`

Elastic CA trusted Fingerprint:

To get this value you need to sign the certificate from ES container

Firtst copy the certificate to a temp folder

```bash
docker cp woovi-es01-1:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/.

```

Then sign the certificate getting the Fingerprint


```
openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g
```


The output will be something like this:

`5A7464CEABC54FA60CAD3BDF16395E69243B827898F5CCC93E5A38B8F78D5E7`

Use the value on the field Elastic CA trusted Fingerprint

Now in Advanced YAML configuration you will need the certificate.

Print the certificate you copied from the container


```
cat /tmp/ca.crt
```



And edit to match the format on this example [file](https://github.com/caiocampoos/elk-local-dev/blob/main/observability/exampl-cat-file.yml)
Its important not to input the wrong indentation, yml really sucks and this step is usually to blame when something goes wrong.


Save and apply settings, wait a few seconds and you should be able to see system metrics like memory and cpu usage on the agent page [here](https://localhost:5601/app/fleet/agents)

That indicates your fleet server is configured the correct way.

Now in your local development you only need to use apm config, go over on the agent configuration for APM in your desired language and check out, most are like the following:

```
{
  serviceName: 'my-service-name',
  secretToken: 'supersecrettoken',
  serverUrl: 'http://localhost:8200',
  environment: 'my-environment'
}

```

Check out for aditional information in this agent integration [page](https://localhost:5601/app/integrations/detail/apm-8.11.2/overview)

We're constantly seeking ways to automate and streamline processes. If you have any ideas on how to enhance this workflow, please don't hesitate to share them with us!

---
Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are [hiring](https://woovi.com/jobs/)!

---