Monitor Kubernetes Events as Logs with Fluent Bit
Tags: kubernetes, tutorial, monitoring, fluentbit
Published: 2023-09-07 09:37:45

Kubernetes event is report of an event happening within the cluster. It tracks any changes made to the cluster which is a good piece of information for debugging.
However, the retention time is limited. 

## Fluentbit 
[Fluent Bit](https://fluentbit.io/) is a fast and lightweight Telemetry agent. We will be using Kubernetes Events input plugin to process the Kubernetes event as log.

## Monitor Kubernetes Event
First we will install Fluentbit using [Helm](https://helm.sh/). We will assume you already have a Kubernetes cluster handy.

We will deploy Fluentbit with a minimal configuration as shown below. Save this as `values.yaml`.

```yaml
rbac:
  eventsAccess: true

config:
  inputs: |
    [INPUT]
        Name kubernetes_events
        Tag k8s_events.*
        kube_url https://kubernetes.default.svc

  outputs: |
    [OUTPUT]
        Name stdout
        Match k8s_events.*

```

We will need `rbac.eventsAccess: true` to give Fluentbit access to Kubernetes Event resource. In `config.inputs`, we define the input of Fluentbit which will be Kubernetes event in this case. And in `config.outputs`, we define where the output of the incoming Kubernetes event goes to.

```sh
$ helm repo add fluent https://fluent.github.io/helm-charts

$ helm repo update

$ helm upgrade --install fluent-bit fluent/fluent-bit -f values.yaml
```

Let's make sure that our Fluentbit is running. 

```sh
 kubectl get po
NAME               READY   STATUS    RESTARTS        AGE
fluent-bit-4dm2l   1/1     Running   0               27h
```

We got the fluentbit running so let's create a sample pod to create some Kubernetes events.

```sh
$ kubectl run nginx --image=nginx -- sleep 1000
```

And if we take a look at the fluent bit log, we can see the events related to nginx pod we just created.

```sh
$ kubectl logs ds/fluent-bit
[1] k8s_events.*: [[1694078558.000000000, {}], {"metadata"=>{"name"=>"nginx.17829379194ff602", "namespace"=>"default", "uid"=>"08314b94-2b02-4485-8a86-65cb07f1b3e1", "resourceVersion"=>"164604", "creationTimestamp"=>"2023-09-07T09:22:38Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:38Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Pulling", "message"=>"Pulling image "nginx"", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:38Z", "lastTimestamp"=>"2023-09-07T09:22:38Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[0] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a6e3d22e3", "namespace"=>"default", "uid"=>"2fe002bc-c1c5-47b5-b057-a0764602893a", "resourceVersion"=>"164611", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Pulled", "message"=>"Successfully pulled image "nginx" in 5.71974592s (5.71977817s including waiting)", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[1] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a6ea78c6c", "namespace"=>"default", "uid"=>"0058a999-0966-4b66-84c8-3361622fc7a8", "resourceVersion"=>"164612", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Created", "message"=>"Created container nginx", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[2] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a735969d5", "namespace"=>"default", "uid"=>"8830ae74-2f2c-4d5f-92a8-6f48aff7de59", "resourceVersion"=>"164615", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Started", "message"=>"Started container nginx", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]

```

If we take a look at the Kubernetes event with kubectl, we can see the same event which we saw in the Fluentbit log for long term storage.

```
$ kubectl get events --field-selector involvedObject.name=nginx                                           [18:31:21]
LAST SEEN   TYPE     REASON      OBJECT      MESSAGE
8m45s       Normal   Scheduled   pod/nginx   Successfully assigned default/nginx to kind-control-plane
8m44s       Normal   Pulling     pod/nginx   Pulling image "nginx"
8m39s       Normal   Pulled      pod/nginx   Successfully pulled image "nginx" in 5.71974592s (5.71977817s including waiting)
8m39s       Normal   Created     pod/nginx   Created container nginx
8m39s       Normal   Started     pod/nginx   Started container nginx
```

We tested out the Kubernetes event input plugin with a simple configuration. From here we can customize the fluentbit configuration to send the logs to third party storages outside of Kubernetes cluster .

## Reference
- https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/event-v1/
- https://docs.fluentbit.io/manual/pipeline/inputs/kubernetes-events