Send Slack Notification when Pod is in crashloopbackoff state
Tags: kubernetes, devops, showdev, opensource
Published: 2023-11-07 10:55:17

[Projectsveltos](https://github.com/projectsveltos) is a Kubernetes add-on controller that simplifies the deployment and management of add-ons and applications across multiple clusters. It runs in the management cluster and can programmatically deploy and manage add-ons and applications on any cluster in the fleet, including the management cluster itself. Sveltos supports a variety of add-on formats, including Helm charts, raw YAML, Kustomize, Carvel ytt, and Jsonnet.

[Projectsveltos](https://github.com/projectsveltos) though goes beyond managing add-ons and applications across a fleet of #Kubernetes Clusters. It can also proactively monitor cluster health and provide real-time notifications.

For example, Sveltos can be configured to detect pods in a crashloopbackoff state within any of the managed clusters and send immediate #slack notifications alerting administrators to potential issues.

## Detect a Pod in crashloopbackoff state

![Detect crashing pods and send Slack Notification](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wzwynpxi16ml7cbgxioi.gif)

Projectsveltos has two custom resource definitions to achieve this goal:

1. __HealthCheck__ defines what to monitor. It accepts a #lua script. Sveltos's monitoring capabilities extend to all Kubernetes resources, including custom resources, ensuring comprehensive oversight of your infrastructure. 
2. __ClusterHealthCheck__ defines which clusters to monitor and where to send notifications

```yaml
apiVersion: lib.projectsveltos.io/v1alpha1
kind: HealthCheck
metadata:
 name: crashing-pod
spec:
 group: ""
 version: v1
 kind: Pod
 script: |
   function evaluate()
     hs = {}
     hs.status = "Healthy"
     hs.ignore = true
     if obj.status.containerStatuses then
        local containerStatuses = obj.status.containerStatuses
        for _, containerStatus in ipairs(containerStatuses) do
          if containerStatus.state.waiting and containerStatus.state.waiting.reason == "CrashLoopBackOff" then
            hs.status = "Degraded"
            hs.ignore = false
            hs.message = obj.metadata.namespace .. "/" .. obj.metadata.name .. ":" .. containerStatus.state.waiting.message
            if containerStatus.lastState.terminated and containerStatus.lastState.terminated.reason then
              hs.message = hs.message .. "\nreason:" .. containerStatus.lastState.terminated.reason
            end
          end
```

```yaml
apiVersion: lib.projectsveltos.io/v1alpha1
kind: ClusterHealthCheck
metadata:
 name: crashing-pod
spec:
 clusterSelector: env=fv
 livenessChecks:
 - name: crashing-pod
   type: HealthCheck
   livenessSourceRef:
     kind: HealthCheck
     apiVersion: lib.projectsveltos.io/v1alpha1
     name: crashing-pod
 notifications:
 - name: slack
   type: Slack
   notificationRef:
     apiVersion: v1
     kind: Secret
     name: slack
     namespace: default
```


![Slack Notifications](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/my2825zopudzm64ykn86.png)

All YAMLs used in this example can be found [here](https://github.com/projectsveltos/demos/tree/main/observability)

## Centralised View
[Projectsveltos](https://github.com/projectsveltos) also offers the ability to display health of the clusters it manages. This information can then be accessed and displayed using Sveltos' CLI in the management cluster. 

![Sveltosctl](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ohyge2hdkdosaft8sty1.png)

## 👏 Support this project
If you enjoyed this article, please check out the [Projectsveltos GitHub repo](https://github.com/projectsveltos). 

You can also [star 🌟 the project](https://github.com/projectsveltos/addon-controller) if you found it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.
Thank you for reading!