Using Projectsveltos to Manage Kubernetes Add-ons on Civo Cloud Clusters
Tags: kubernetes, showdev, opensource, sre
Published: 2023-11-02 13:16:18

[Civo Cloud](https://www.civo.com/) is a cloud computing platform that provides developers with a simple and predictable way to deploy and manage their applications. Civo is focused on providing a developer-friendly experience, with a focus on Kubernetes and other cloud-native technologies.

[Projectsveltos](https://github.com/projectsveltos) is a Kubernetes add-on controller that simplifies the deployment and management of add-ons and applications across multiple clusters. It runs in the management cluster and can programmatically deploy and manage add-ons and applications on any cluster in the fleet, including the management cluster itself. Sveltos supports a variety of add-on formats, including Helm charts, raw YAML, Kustomize, Carvel ytt, and Jsonnet.

[Projectsveltos](https://github.com/projectsveltos) allows add-ons and applications to be represented as templates. Before deploying to managed clusters, Sveltos instantiates these templates. Sveltos can gather the information required to instantiate the templates from either the management cluster or the managed clusters themselves.

## Create Kubernetes clusters on Civo cloud

Deploying Kubernetes clusters on Civo Cloud was a breeze. In my case, I only needed to specify the number of nodes and the Kubernetes version, and within a few minutes, all clusters were up and running.



![Civo Kubernetes clusters](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zxu9dw7nds9l0e1x3o6o.png)


## Register cluster to be managed by Projectsveltos

Using one of the Kubernetes clusters as the management cluster, I deployed Projectsveltos. The management cluster is used by Projectsveltos to [manage add-ons and applications](https://projectsveltos.github.io/sveltos/configuration/) in other Kubernetes clusters.

To deploy Projectsveltos, I pointed kubectl to the management cluster and ran the following commands:


```
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/default-classifier.yaml
```

Once Projectsveltos was deployed, I easily registered the other Kubernetes clusters with it using [sveltosctl](https://github.com/projectsveltos/sveltosctl). I still pointed kubectl to the management cluster and ran the following commands:

```
sveltosctl register cluster --namespace=eng --cluster=civo --kubeconfig=<civo-eng-kubeconfig>
sveltosctl register cluster --namespace=qe --cluster=civo --kubeconfig=<civo-qe-kubeconfig>
sveltosctl register cluster --namespace=marketing --cluster=civo --kubeconfig=<civo-marketing-kubeconfig>
```

Each of these commands created a [SveltosCluster](https://raw.githubusercontent.com/projectsveltos/libsveltos/main/api/v1alpha1/sveltoscluster_type.go) instance in the management cluster. Each SveltosCluster instance represents a Kubernetes cluster managed by Projectsveltos.

I then added labels to the clusters:

```
kubectl get sveltoscluster -A --show-labels
NAMESPACE   NAME   READY   VERSION        LABELS
eng         civo   true    v1.28.2+k3s1   department=eng,env=production,sveltos-agent=present
marketing   civo   true    v1.28.2+k3s1   department=marketing,env=production,sveltos-agent=present
qe          civo   true    v1.28.2+k3s1   department=qe,env=production,sveltos-agent=present
```

## Deploy add-ons
Using Projectsveltos in the management cluster, we created ClusterProfile instances to specify the add-ons and applications to deploy to each cluster.

We created a [ClusterProfile](https://github.com/projectsveltos/addon-controller/tree/main/examples) for Kyverno to deploy it to all production clusters.

```yaml
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: kyverno
spec:
  clusterSelector: env=production
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v3.0.3
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install
    values: |
      admissionController:
        replicas: 3
```

We created two ClusterProfiles, one for Prometheus and Grafana, and one for Kubevela, to deploy them to the QE cluster.

```yaml
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: prometheus-grafana
spec:
  clusterSelector: department=qe
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install
```

```yaml
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile 
metadata: 
  name: kubevela-core
spec:
  clusterSelector: department=qe
  syncMode: Continuous
  helmCharts:
  - repositoryURL: https://kubevela.github.io/charts
    repositoryName: kubevela
    chartName: kubevela/vela-core
    chartVersion: 1.9.6
    releaseName: kubevela-core-latest
    releaseNamespace: vela-system
    helmChartAction: Install
```

We created a ClusterProfile for Nginx to deploy it to the engineering cluster.

```yaml
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: nginx
spec:
  clusterSelector: department=eng
  helmCharts:
  - repositoryURL:    https://helm.nginx.com/stable/
    repositoryName:   nginx-stable
    chartName:        nginx-stable/nginx-ingress
    chartVersion:     0.14.0
    releaseName:      nginx-latest
    releaseNamespace: nginx
    helmChartAction:  Install
```

To get a summary of what was deployed to each cluster, we used [sveltosctl](https://github.com/projectsveltos/sveltosctl) with kubeconfig pointing to the management cluster:

```
sveltosctl show addons  
+----------------+---------------+-------------+----------------------+---------+-------------------------------+--------------------+
|    CLUSTER     | RESOURCE TYPE |  NAMESPACE  |         NAME         | VERSION |             TIME              |  CLUSTER PROFILES  |
+----------------+---------------+-------------+----------------------+---------+-------------------------------+--------------------+
| eng/civo       | helm chart    | kyverno     | kyverno-latest       | 3.0.3   | 2023-11-01 14:19:52 +0100 CET | kyverno            |
| eng/civo       | helm chart    | nginx       | nginx-latest         | 0.14.0  | 2023-11-01 14:20:39 +0100 CET | nginx              |
| qe/civo        | helm chart    | kyverno     | kyverno-latest       | 3.0.3   | 2023-11-01 14:19:46 +0100 CET | kyverno            |
| qe/civo        | helm chart    | vela-system | kubevela-core-latest | 1.9.6   | 2023-11-01 14:22:38 +0100 CET | kubevela-core      |
| qe/civo        | helm chart    | prometheus  | prometheus           | 23.4.0  | 2023-11-01 14:23:14 +0100 CET | prometheus-grafana |
| qe/civo        | helm chart    | grafana     | grafana              | 6.58.9  | 2023-11-01 14:23:17 +0100 CET | prometheus-grafana |
| marketing/civo | helm chart    | kyverno     | kyverno-latest       | 3.0.3   | 2023-11-01 14:20:01 +0100 CET | kyverno            |
+----------------+---------------+-------------+----------------------+---------+-------------------------------+--------------------+ 
```


![Managing add-ons in a fleet of Civo Kubernetes Clusters](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pjutzbziojnyr7pej1b5.gif)

## 👏 Support this project
If you enjoyed this article, please check out the [Projectsveltos GitHub repo](https://github.com/projectsveltos). 

You can also [star 🌟 the project](https://github.com/projectsveltos/addon-controller) if you found it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.

Thank you for reading!