Logz.io: collection logs from Kubernetes — fluentd vs filebeat
Tags: monitoring, kubernetes
Published: 2021-02-25 14:14:36

---
title: Logz.io: collection logs from Kubernetes — fluentd vs filebeat
published: true
date: 2021-02-01 07:58:53 UTC
tags: monitoring,kubernetes
canonical_url: https://rtfm.co.ua/en/logz-io-collection-logs-from-kubernetes-fluentd-vs-filebeat/
---

### Logz.io: collection logs from Kubernetes — fluentd vs filebeat

![](https://cdn-images-1.medium.com/max/1024/1*P8PaFnT6TshOzcjBylRo_g.png)

We are using [Logz.io](https://logz.io/) to collect our Kubernetes cluster logs (also, there is a local [Loki](https://rtfm.co.ua/en/grafana-labs-loki-logs-collector-and-monitoring-system/) instance).

Logs are collected and processed by a [Fluentd](https://www.fluentd.org/) pod on every WorkerNode which are deployed from a [DaemonSet](https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac.yaml) in its default configuration, see the documentation here — [logzio-k8s](https://github.com/logzio/logzio-k8s).

The problem we faced is that those pods are consuming too much CPU — up to 3000 millicpu, while our WorkerNodes has only 4 cores, e.g. 4000 millicpu.

So, to solve this issue I’ve decided to search for similar log collectors and the second thing to do is was to able to deploy them with Ansible via a Helm chart, but Fluentd has no ready charts yet.

The found solution w a Filebeat collector, see [Ship k8s logs with Helm via Filebeat](https://docs.logz.io/shipping/log-sources/k8s-over-helm.html) — let’s try it.

At first, we will deploy it manually do a Dev cluster, and then we’ll add an Aisible task to deploy it on our Production.

### Logz.io Filebeat Helm chart

Add a repository:

```
$ helm repo add logzio-helm [https://logzio.github.io/logzio-helm/filebeat](https://logzio.github.io/logzio-helm/filebeat)
“logzio-helm” has been added to your repositories
```

Next, we need to fin our token and region — go to the account’s General Settings:

![](https://cdn-images-1.medium.com/max/655/0*SRB0C7oof0k8matG.png)

As we have the default region — us-east-1, see [How to look up your account region](https://docs.logz.io/user-guide/accounts/account-region.html), then we can remove the `secrets.logzioRegion` from the Helm chart's arguments.

Also, by default Logz.io client will be installed to the `kube-system` namespace, but I'd like to place it to a dedicated NS to make it easier to monitor its used resources.

The Namespace is configured in the [values](https://github.com/logzio/logzio-helm/blob/master/filebeat/values.yaml#L14) file so let’s override it with `--set`, plus let's add the `--create-namespace` and `--debug` options:

```
$ helm install — namespace=dev-1–18-devops-logzio-ns \
 --create-namespace — debug \
 --set secrets.logzioShippingToken='AVG***Onq' \
 --set secrets.clusterName='bttrm-eks-dev-1–18' \
 --set namespace=dev-1–18-devops-logzio-ns \
logzio-k8s-logs logzio-helm/logzio-k8s-logs
```

Check pods:

```
$ kubectl -n dev-1–18-devops-logzio-ns get pod
NAME READY STATUS RESTARTS AGE
filebeat-2qt5s 1/1 Running 0 96s
filebeat-4xb44 1/1 Running 0 96s
filebeat-9prr9 1/1 Running 0 96s
filebeat-cth47 1/1 Running 0 96s
filebeat-fgmgx 1/1 Running 0 96s
filebeat-gb5ts 1/1 Running 0 96s
filebeat-hs9tr 1/1 Running 0 96s
filebeat-nskvg 1/1 Running 0 96s
filebeat-wfgbg 1/1 Running 0 96s
```

And logs in the Kibana on the [app.logs.io](https://app.logz.io/) page:

![](https://cdn-images-1.medium.com/max/1024/0*xdqsDKaFybGudO7d.png)

### Ansible

There is a post about deploying a Helm chart with Ansible — [Ansible: модуль community.kubernetes и установка Helm-чарта с ExternalDNS](https://rtfm.co.ua/ansible-modul-community-kubernetes-i-ustanovka-helm-charta-s-externaldns/) (_Rus_), at this time just a quick example.

Add a when condition as we will deploy Logz.io to only the Production cluster:

```
- name: "Add Logzio chart repo"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm_repository:
    name: "logzio-helm"
    repo_url: "https://logzio.github.io/logzio-helm/filebeat"

- name: "Deploy Logzio Filebit chart to the {{ eks_env }}-devops-logzio-ns namespace"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm:
    kubeconfig: "{{ kube_config_path }}"
    name: "logzio-k8s-logs"
    chart_ref: "logzio-helm/logzio-k8s-logs"
    release_namespace: "{{ eks_env }}-devops-logzio-ns"
    create_namespace: true
    values:
      secrets:
        logzioShippingToken: "{{ logzio_token }}"
        clusterName: "{{ eks_cluster_name }}"
      namespace: "{{ eks_env }}-devops-logzio-ns"
```

Specify the `logzio_token` in the Ansible variables and encrypt it with the [`ansible-vault`](https://rtfm.co.ua/ansible-ispolzovanie-vault-zashifrovannogo-xranilishha/), as we have our Ansible roles stored in our Github repository.

### Fluentd vs Filebeat — CPU and performance

And finally CPU usage: old `fluentd` (Ruby + C) on the left side vs new `filebeat` (Golang) at the right side:

![](https://cdn-images-1.medium.com/max/1024/0*ETbJU7SNO0_lKEek.png)

_Originally published at_ [_RTFM: Linux, DevOps and system administration_](https://rtfm.co.ua/en/logz-io-collection-logs-from-kubernetes-fluentd-vs-filebeat/)_._

* * *