5-Step Approach: Projectsveltos for Kubernetes add-on deployment and management on RKE2
Tags: kubernetes, opensource, devops, tutorial
Published: 2023-12-18 12:40:57

Working with many different Kubernetes add-on deployments, the actual deployment and management of those across different clusters, on-prem and in the Cloud, can be challenging and sometimes frustrating.

[Projectsveltos](https://github.com/projectsveltos) is a Kubernetes add-on controller that simplifies the deployment and management of different add-ons and applications across multiple clusters (on-prem, Cloud). Sveltos runs in a management cluster and programmatically deploys and manages add-ons and applications on any cluster in the fleet, including the management cluster itself. Sveltos supports many add-on formats, including Helm charts, raw YAML/JSON, Kustomize, Carvel ytt, and Jsonnet.

In this blog post, we will demonstrate how easy and fast it is to deploy Sveltos on an [RKE2](https://docs.rke2.io/) cluster with the help of [ArgoCD](https://github.com/argoproj/argo-cd), register two RKE2 Cluster API ([CAPI](https://github.com/kubernetes-sigs/cluster-api)) clusters and create a ClusterProfile to deploy Prometheus and Grafana Helm charts down the managed CAPI clusters.

### Diagram

![Projectsveltos Demo Diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6bl643bnnwzxj7821f38.jpg)

## Prerequisites

For this demonstration, I have already installed ArgoCD on a central cluster. If you would like to learn more about the ArgoCD installation, go through the official documentation found [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). If you would like to follow along, below you can find the lab details used.

```
- - - - - -+ - - - - - - - - - - - + - - - - - - - - - - -+
| Cluster Name |      Type         |      Version         |
+ - - - - - - -+ - - - - - - - - - - + - - - - - - - - - -+
| cluster04 | Management Cluster   | RKE2 v1.26.11+rke2r1 |
| cluster12 | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1  |
| cluster13 | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1  |
+ - - - - - -+ - - - - - - - - - + - - - - - - - - - - - -+
```

## Step 1: Deploy Sveltos as a Helm Chart cluster04

Sveltos can be deployed either as a **manifest** or as a **Helm chart** . For more information about the different installation options, check the link [here]( https://projectsveltos.github.io/sveltos/install/install/). In my case, I chose to follow the **GitOps** approach and let ArgoCD deal with the comparison and synchronisation of the Git repository where the code to deploy Sveltos is stored, with the actual running application.

If you are unsure how to deploy Helm charts with ArgoCD, have a look [here](https://argo-cd.readthedocs.io/en/stable/user-guide/helm/).

### Verification

After we deploy Sveltos, we want to ensure everything is in a working and fully functional state. This can be achieved either from the ArgoCD UI or from the management cluster itself.

![ArgoCD - Sveltos Helm Chart Deployment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/09zc1946oqzo2qm6vyf5.png)

```
$ kubectl get pods -n projectsveltos

NAME                                        READY   STATUS    RESTARTS   AGE
access-manager-77c7c64477-ns8ml             2/2     Running   0          70s
addon-compliance-manager-7f449d884c-6kgqr   2/2     Running   0          69s
addon-controller-55d7d848ff-ps8l8           2/2     Running   0          70s
classifier-manager-67d6f67d5b-cgpr7         2/2     Running   0          70s
event-manager-69db45b65d-htz5l              2/2     Running   0          70s
hc-manager-5679c69dcc-z6s48                 2/2     Running   0          70s
sc-manager-84dbd64fb4-6hwpf                 2/2     Running   0          70s
shard-controller-56678bcf8c-zjbvc           2/2     Running   0          70s
```

## Step 2: Install the Sveltosctl

The [Sveltosctl](https://github.com/projectsveltos/sveltosctl), is the command-line interface (CLI) for Sveltos. This is an available option to query Sveltos resources and it is available as a [Kubernetes pod](https://projectsveltos.github.io/sveltos/install/sveltosctl/) or as a [binary](https://github.com/projectsveltos/sveltosctl/releases).

As I would like to register cluster12 and cluster13 to the Sveltos management cluster, the Sveltosctl as a binary will be used.

## Step 3: Register CAPI Clusters with Sveltos

To register any cluster with Sveltos, you only need three things:

1. A ServiceAccount for Sveltos and a kubeconfig associated with that account;
2. A namespace where you want to register the external cluster;
3. The **Sveltosctl** should **point** to the **management cluster** and then perform the *'sveltosctl register cluster'* command.

Now, if you are unsure how to create a Service Account and an associated kubeconfig, do not worry. There is a script publicly available to create everything you need automatically.

### Registration

```
$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster12 --kubeconfig=cluster12.yaml

$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster13 --kubeconfig=cluster13.yaml
```

From the commands above, we register cluster12 and cluster13 in the namespace projectsveltos. Of course, you can register the clusters to a namespace of your preference.

### Verification
```
$ kubectl get sveltosclusters -n projectsveltos

NAME        READY   VERSION
cluster12   true    v1.26.6+rke2r1
cluster13   true    v1.26.6+rke2r1
```

## Step 4: Cluster Labelling

To allow Sveltos to deploy and manage Kubernetes add-ons, the concept of [ClusterProfile](https://github.com/projectsveltos/sveltos-manager/blob/main/api/v1alpha1/clusterprofile_types.go) and cluster labelling comes into play. ClusterProfile is the *CustomerResourceDefinition* used to instruct Sveltos which add-ons to deploy on a set of clusters.

For this demonstration, we will set the label "env:prod" to both Sveltos clusters. The below commands are executed on the **management cluster** (cluster04).

```
$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   sveltos-agent=present
```

```
$ kubectl label sveltosclusters cluster12 env=prod -n projectsveltos

$ kubectl label sveltosclusters cluster13 env=prod -n projectsveltos
```

```
$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present
```

## Step 5: ClusterProfile for Grafana and Prometheus

The below ClusterProfile is an example of a Helm chart deployment of Grafana and Prometheus to Sveltos clusters with the label set to *"env:prod"*.

```
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: prometheus-grafana
spec:
  clusterSelector: env=prod
  helmCharts:
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install
```

### Apply the ClusterProfile

```
$ kubectl apply -f "grafana_prometheus.yaml"
```
Once the ClusterProfile is applied to the **management cluster**, the expected result is to have the Grafana and the Prometheus deployment on both managed clusters.

### Verification

```
$ sveltosctl show addons

+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
|         CLUSTER          | RESOURCE TYPE | NAMESPACE  |    NAME    | VERSION |             TIME              |  CLUSTER PROFILES  |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster12 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:20 +0100 CET | prometheus-grafana |
| projectsveltos/cluster12 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:23 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
```

### Verification - Cluster12
```
$ kubectl get pods -n grafana

NAME                           READY   STATUS    RESTARTS   AGE
pod/grafana-78764f9cd6-zsqdx   1/1     Running   0          81s

$ kubectl pods all -n prometheus

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/prometheus-alertmanager-0                            1/1     Running   0          2m3s
pod/prometheus-kube-state-metrics-587bd996f6-l94zq       1/1     Running   0          2m3s
pod/prometheus-prometheus-node-exporter-khw75            1/1     Running   0          2m3s
pod/prometheus-prometheus-pushgateway-75986b9c9f-2ql7v   1/1     Running   0          2m3s
pod/prometheus-server-86c66b89c6-7xk9r                   2/2     Running   0          2m3s
```
The same verification can be performed for cluster13.

### Remove Label 'env:prod' cluster12

You might wonder what will happen if we remove the label 'env:prod' from either cluster12 or cluster13. The answer is that Sveltos will identify the missing label 'env:prod' and undeploy the Grafana and the Prometheus deployment from the cluster.

Let's have a look.

### Remove Label
```
$ kubectl label sveltosclusters cluster12 env- -n projectsveltos
```

### Verification
```
$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present
```
```
sveltosctl show addons

+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
|         CLUSTER          | RESOURCE TYPE | NAMESPACE  |    NAME    | VERSION |             TIME              |  CLUSTER PROFILES  |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster13 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
```

As expected, Sveltos removed the deployments. The same will happen if we register a new cluster and assign the label 'env:prod'. Sveltos will take care of the complete lifecycle of your Kubernetes deployments in a simple and straightforward manner.


##👏 Support this project

Every contribution counts! If you enjoyed this article, check out the Projectsveltos [GitHub](https://github.com/projectsveltos) repo. You can star 🌟 the project if you found it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!