Managed cluster health checks
Tags: showdev, kubernetes, lua, management
Published: 2023-03-21 17:06:49

[Sveltos](https://github.com/projectsveltos) is a project to deploy Kubernetes add-ons in tens of managed clusters.

We have already covered few aspects on Sveltos in different posts:

1. [Kubernetes add-ons deployment](https://dev.to/gianlucam76/deploy-kubernetes-add-ons-in-clusterapi-powered-cluster-7gm);
2. [Upgrading add-ons as cluster runtime state changes](https://dev.to/gianlucam76/deploy-kubernetes-addons-from-management-cluster-5j2);
3. [Multi-tenancy with Sveltos](https://dev.to/gianlucam76/kubernetes-multi-tenancy-with-projectsveltos-4o0k).

In this post, we will cover how to instruct Sveltos to continuously evaluate managed cluster health statuses and send notifications any time health changes.

## Add-on deployment state

When a [ClusterProfile](https://github.com/projectsveltos/sveltos-manager/blob/main/api/v1alpha1/clusterprofile_types.go) instance is created, Sveltos starts watching for clusters matching ClusterProfile clusterSelector field. In each matching workload cluster, Sveltos deploys all referenced add-ons (helm charts and/or Kubernetes resource YAMLs).

So, as first use case, it comes natural to have Sveltos sends notifications when all add-ons are deployed.

The notifications fields is a list of all notifications to be sent when liveness check states change.

![Sveltos ClusterHealthCheck CRD](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/14csx90r0jaxcc8sfauf.png)

where slack secret contains Slack channel and token

```
kubectl create secret generic slack --from-literal=SLACK_TOKEN=<your token> --from-literal=SLACK_CHANNEL_ID=<your channel id> --type=addons.projectsveltos.io/cluster-profile 
```

where webex secret contains Webex room and token

```
kubectl create secret generic slack --from-literal=WEBEX_TOKEN=<your token> --from-literal=WEBEX_ROOM_ID=<your channel id> --type=addons.projectsveltos.io/cluster-profile 

```

Posting this _ClusterHealthCheck_ instance, when add-ons are deployed in any cluster matching the clusterSelector, Sveltos:

1. generates a Kubernetes event;
2. send a slack message;
3. send a webex message.

## Custom Health Checks

Is that enough though? The answer clearly is no. Once all add-ons are deployed, we want to have a generic mechanism to assess managed cluster healths based on the state of Kubernetes resources deployed in those clusters.

Sveltos has a CRD, [HealthCheck](https://github.com/projectsveltos/libsveltos/blob/main/api/v1alpha1/healthcheck_type.go) to define custom health checks. Sveltos supports custom health checks written in [Lua](https://www.lua.org/).

HealthCheck Spec section contains following fields:

1. _Spec.Group_/_Spec.Version_/_Spec.Kind_ fields indicating which Kubernetes resources the HealthCheck is for. Sveltos will watch and evaluate those resources anytime a change happens;
1. _Spec.Namespace_ field can be used to filter resources by namespace;
1. _Spec.LabelFilters_ field can be used to filter resources by labels;
1. _Spec.Script_ can contain a Lua script, which define a custom health check.

When providing Sveltos with a Lua script, Sveltos expects following format:

1. must contain a function _function evaluate()_. This is the function that is directly invoked and passed a Kubernetes resource (inside the function _obj_ represents the passed in Kubernetes resource);
2. must return a Lua table with following fields:
status: which can be set to either one of _Healthy_/_Progressing_/_Degraded_/_Suspended_;
3. ignore: this is a bool indicating whether Sveltos should ignore this resource. If hs.ignore is set to true, Sveltos will ignore the resource causing that result;
4. message: this is a string that can be set and Sveltos will print if set.


![Sveltos HealthCheck instance](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c9gmjbxb9uldh3auwz5s.png)

In above example, we are creating an HealthCheck that watches for all ConfigMaps. _hs_ is the health status object we will return to Sveltos. It must contain a status attribute which indicates whether the resource is _Healthy_, _Progressing_, _Degraded_ or _Suspended_. 

By default we set it to Healthy and _hs.ignore=true_, since we don’t want to mess with the status of other, non-OPA ConfigMaps. Optionally, the health status object may also contain a message.

Identify if the ConfigMap is indeed an OPA policy or another kind of ConfigMap. If it is a OPA policy, retrieve the value of the openpolicyagent.org/policy-status annotation. The annotation is set to {"status":"ok"} if the policy was loaded successfully, if errors occurred during loading (e.g., because the policy contained a syntax error) the cause will be reported in the annotation. Depending on the value of the annotation, we set the status and message attributes appropriately.

ClusterHealthCheck instance can reference one or more HealthChecks. A sveltos agent running in each managed cluster keeps evaluating each HealthCheck instance and reports status back to the management cluster any time a change happens.

Sveltos counter-part running in the management cluster, evaluates those reports and sends notification(s) when cluster health state changes.

![ClusterHealthCheck instance referencing HealthChecks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/orxuhqtuppwu705zn7oq.png)