5-Step Approach: ProjectSveltos Event Framework for Kubernetes Deployment with Cilium Gateway API
Tags: kubernetes, opensource, tutorial, eventdriven
Published: 2024-02-19 15:39:44

## Introduction

In the [previous post](https://medium.com/@eleni.grosdouli/argocd-deployment-on-rke2-with-cilium-gateway-api-ab1769cc28a3), we demonstrated how easy it is to deploy ArgoCD with the [Cilium Gateway API](https://docs.cilium.io/en/v1.14/network/servicemesh/gateway-api/gateway-api/) and move away from the Kubernetes Ingress. In today’s post, we will explore how the [Projectsveltos](https://github.com/projectsveltos) [Event Framework](https://projectsveltos.github.io/sveltos/events/addon_event_deployment/) can be utilised to seamlessly perform the same deployment based on an event.

The Sveltos Event Framework was designed to automate the deployment of Kubernetes add-ons triggered on specific cluster events. This functionality proves invaluable in managing environments spanning across multiple clusters.

Sveltos supports an event-driven workflow with the below steps:

1. Define what the event is
2. Select the clusters to watch for such events
3. Define an event trigger. Which add-ons/applications to deploy when the events occur

Once we identify the event we want Sveltos to watch for, we can deploy the **EventSource** and **EventTrigger** Kubernetes CRDs (Custom Resource Definitions). More information about these resources can be found later on.

## Diagram

![Sveltos Event Framework
](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lkyh5wt3ufk8fxemf3m2.png)

## Prerequisites

To follow along, it is strongly advised to take a look at a previous [post](https://medium.com/@eleni.grosdouli/5-step-approach-projectsveltos-for-kubernetes-add-on-deployment-and-management-on-rke2-be3ba7acb24f) detailing the deployment of Sveltos on a cluster. To install Sveltos, just follow the instructions outlined in steps 1 and 2 of the mentioned post. Additionally, as we will use the Cilium Gateway API, have a look at the previous [post](https://medium.com/@eleni.grosdouli/argocd-deployment-on-rke2-with-cilium-gateway-api-ab1769cc28a3) to enable this capability on an RKE2 cluster.

## Lab Setup

```
— — — — — -+ — — — — — — — — — — + — — — — — — — — — — — — -+
| Cluster Name | Type | Version |
+ — — — — — — -+ — — — — — — — — — — + — — — — — — — — — — -+
| rke2-mgmt01 | Management Cluster | RKE2 v1.26.12+rke2r1 |
| rke2-sles-demo | Managed Cluster | RKE2 v1.26.12+rke2r1|
+ — — — — — -+ — — — — — — — — — — + — — — — — — — — — — — — -+

- — — — — — -+ — — — — -+
| Deployment | Version |
+ — — — — — — -+ — — — — -+
| ArgoCD | v2.9.3 |
| Cilium | v1.14.5 |
| GatewayAPI | v0.7.0 |
+ — — — — — — -+ — — — — -+
```

## Step 1: Register and label the RKE2 cluster with Sveltos

Once the RKE2 cluster is in a “Running” state, we will use the [sveltosctl](https://github.com/projectsveltos/sveltosctl) to register it. For the registration, we need three things: a service account, a kubeconfig associated with that account and a namespace. If you are unsure how to create a Service Account and an associated kubeconfig, there is a [script](https://raw.githubusercontent.com/gianlucam76/scripts/master/get-kubeconfig.sh) publicly available to help you out.

### Registration

```bash
$ sveltosctl register cluster --namespace=projectsveltos --cluster=rke2-sles-demo --kubeconfig=rke2-sles-demo.yaml
```

### Verification

```bash
$ kubectl get sveltoscluster -n projectsveltos
NAME               READY   VERSION
rke2-sles-demo   true    v1.26.12+rke2r1
```

### Assing Label

```bash
$ kubectl label sveltoscluster rke2-sles-demo env=staging -n projectsveltos
```

### Verification

```bash
$ kubectl get sveltoscluster -n projectsveltos --show-labels
NAME             READY   VERSION           LABELS
rke2-sles-demo   true    v1.26.12+rke2r1   env=staging,sveltos-agent=present
```

For the demonstration, we will use the label ‘env=staging’ to specify the cluster we want to deploy the Gateway and the HTTPRoute resources.

## Step 2: Pre-Work (management cluster)

Before we move on with the Gateway API implementation, we need to create additional Kubernetes resources for the managed cluster (rke2-sles-demo).

- ArgoCD namespace
- ArgoCD TLS Secret
- Cilium IP Pool
- Cilium GatewayClass
- Gateway
- HTTPRoute

To utilise the power of Sveltos, we will create the above resources in the management cluster as a Secret and configMap resource and use the Sveltos ClusterProfile to deploy them to the managed cluster.

More information on how to create the above resources can be found [here](https://projectsveltos.github.io/sveltos/addons/raw_yaml/).

### ArgoCD Namespace
If the namespace does not exist in the managed cluster, Sveltos will create it!

### ArgoCD TLS Secret

```yaml
apiVersion: v1
data:
  tls.crt: {base64 encoded .crt}
  tls.key: {base64 encoded .key}
kind: Secret
metadata:
  name: argocd-server-tls
  namespace: argocd
type: kubernetes.io/tls
```

```bash
$ kubectl create secret generic argocd-server-tls --from-file=argocd_server_tls.yaml --type=addons.projectsveltos.io/cluster-profile
```

### Cilium IP Pool

```yaml
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "rke2-pool"
spec:
  cidrs:
  - cidr: "10.10.10.0/24"

```

```bash
$ kubectl create configmap ipampool --from-file=ipam-pool.yaml
```

### Cilium GatewayClass

```yaml
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller
```

```bash
$ kubectl create configmap gatewayclass --from-file=gatewayclass.yaml
```

### Gateway

```yaml
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: argocd
  namespace: argocd
spec:
  gatewayClassName: cilium
  listeners:
  - hostname: argocd.example.com
    name: argocd-example-com-http
    port: 80
    protocol: HTTP
  - hostname: argocd.example.com
    name: argocd-example-com-https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - kind: Secret
        name: argocd-server-tls
```

```bash
$ kubectl create configmap gateway --from-file=gateway.yaml
```

### HTTPRoute

```yaml
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  creationTimestamp: null
  name: argocd
  namespace: argocd
spec:
  hostnames:
  - argocd.example.com
  parentRefs:
  - name: argocd
  rules:
  - backendRefs:
    - name: argocd-server
      port: 80
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents: []
```

```bash
$ kubectl create configmap gateway --from-file=httproute.yaml
```

### Verification

```bash
$ kubectl get secret
NAME                TYPE                                       DATA   AGE
argocd-server-tls   addons.projectsveltos.io/cluster-profile   1      20m

$ kubectl get cm
NAME               DATA   AGE
gateway            1      35s
gatewayclass       1      20m
httproute          1      3s
ipam-pool          1      22m
kube-root-ca.crt   1      141m
```

### ClusterProfile

```yaml
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
 name: cilium-gateway-api
spec:
 clusterSelector: env=staging
 syncMode: Continuous
 policyRefs:
 - name: ipampool
   namespace: default
   kind: ConfigMap
 - name: gatewayclass
   namespace: default
   kind: ConfigMap
 - name: argocd-server-tls
   namespace: default
   kind: Secret
```

The ClusterProfile definition will match the managed cluster with the label selector set to ‘env=staging’ and deploy the Cilium IP Pool, the GatewayClass and the ArgoCD TLS secret.

### Apply ClusterProfile
```bash
$ kubectl apply -f "clusterprofile.yaml"
```

### Verification

```bash
$ sveltosctl show addons

+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
|            CLUSTER            |             RESOURCE TYPE              | NAMESPACE |       NAME        | VERSION |             TIME              |                  PROFILES                   |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/rke2-sles-demo | cilium.io:CiliumLoadBalancerIPPool     |           | rke2-pool         | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:GatewayClass |           | cilium            | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | :Secret                                | argocd    | argocd-server-tls | N/A     | 2024-02-11 17:39:58 +0100 CET | ClusterProfile/cilium-gateway-api           |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
```

## Step 3: Define the Event Framework (management cluster)

As mentioned at the beginning of the post, we will use the **EventSource** to define which events Sveltos will watch for and the **EventTrigger** resource to perform actions based on an event.

### EventSource

```yaml
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventSource
metadata:
 name: http-https-service
spec:
 collectResources: true
 resourceSelectors:
 - group: ""
   version: "v1"
   kind: "Service"
   namespace: argocd
   evaluate: |
     function evaluate()
       hs = {}
       hs.matching = false
       if obj.spec.ports ~= nil then
         for _,p in pairs(obj.spec.ports) do
           if p.port == 80 or p.port == 443 then
             hs.matching = true
           end
         end
       end
       return hs
     end
```
The EventSource uses the [Lua](https://www.lua.org/) language to search for any services with ports set to 80 or 443 in the ‘argocd’ namespace. More examples can be found [here](https://projectsveltos.github.io/sveltos/events/addon_event_deployment/).

### EventTrigger

```yaml
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: EventTrigger
metadata:
 name: service-network-policy
spec:
 sourceClusterSelector: env=staging
 eventSourceName: http-https-service
 oneForEvent: true
 policyRefs:
 - name: httproute
   namespace: default
   kind: ConfigMap
 - name: gateway  
   namespace: default
   kind: ConfigMap
```

### Apply Resources

```bash
$ kubectl apply -f "eventsource.yaml"
eventsource.lib.projectsveltos.io/loadbalancer-service created

$ kubectl apply -f "eventtrigger.yaml"
eventtrigger.lib.projectsveltos.io/service-network-policy created
```

Once an event is spotted by Sveltos, the HTTPRoute and Gateway Kubernetes add-ons will get deployed on the managed cluster with the label selector set to ‘env=staging’. In our case, the cluster ‘rke2-sles-demo’ will match the ClusterSelector and will automatically get the specified resources.

## Step 4: Deploy ArgoCD Manifests (managed cluster)
Now, it is time to deploy the ArgoCD manifests and wait Sveltos to **notice the event** and **trigger** the **deployment** of the HTTPRoute and Gateway resources in the ‘argocd’ namespace.

```bash
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```

## Step 5: Verification and Tests

As we have installed the ArgoCD manifests, the ‘argocd-server’ service should have been created in the ‘argocd’ namespace. That means the Sveltos EventSource should have been activated and triggered the deployment of the HTTPRoute and Gateway Kubernetes resources.

### Sveltos Verification

```bash

$ sveltosctl show addons
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
|            CLUSTER            |             RESOURCE TYPE              | NAMESPACE |       NAME        | VERSION |             TIME              |                  PROFILES                   |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/rke2-sles-demo | cilium.io:CiliumLoadBalancerIPPool     |           | rke2-pool         | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:GatewayClass |           | cilium            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | :Secret                                | argocd    | argocd-server-tls | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/cilium-gateway-api           |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:HTTPRoute    | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-1lbzxnp4cs96gpo38g8b |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:Gateway      | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-1lbzxnp4cs96gpo38g8b |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:HTTPRoute    | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-i0nmkq3ebsvpis8ubd3s |
| projectsveltos/rke2-sles-demo | gateway.networking.k8s.io:Gateway      | argocd    | argocd            | N/A     | 2024-02-11 19:55:52 +0100 CET | ClusterProfile/sveltos-i0nmkq3ebsvpis8ubd3s |
+-------------------------------+----------------------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+

$ sveltosctl show usage
+----------------+--------------------+------------------------------+-------------------------------+
| RESOURCE KIND  | RESOURCE NAMESPACE |        RESOURCE NAME         |           CLUSTERS            |
+----------------+--------------------+------------------------------+-------------------------------+
| ClusterProfile |                    | cilium-gateway-api           | projectsveltos/rke2-sles-demo |
| ClusterProfile |                    | sveltos-fvc3vr84yqusfzwpobcl | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | gateway                      | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | ipampool                     | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | gatewayclass                 | projectsveltos/rke2-sles-demo |
| ConfigMap      | default            | httproute                    | projectsveltos/rke2-sles-demo |
+----------------+--------------------+------------------------------+-------------------------------+

```

### Managed Cluster Verification

```bash
$ kubectl get gateway,httproute -n argocd
NAME                                       CLASS    ADDRESS        PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/argocd   cilium   10.10.10.147   True         78s

NAME                                         HOSTNAMES                AGE
httproute.gateway.networking.k8s.io/argocd   ["argocd.example.com"]   78s
```

### Test ArgoCD Accessibility

```bash
curl -ki https://argocd.example.com
HTTP/1.1 200 OK
accept-ranges: bytes
content-length: 788
content-security-policy: frame-ancestors 'self';
content-type: text/html; charset=utf-8
vary: Accept-Encoding
x-frame-options: sameorigin
x-xss-protection: 1
date: Sun, 11 Feb 2024 18:59:57 GMT
x-envoy-upstream-service-time: 0
server: envoy

<!doctype html><html lang="en"><head><meta charset="UTF-8"><title>Argo CD</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32"/><link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16"/><link href="assets/fonts.css" rel="stylesheet"><script defer="defer" src="main.9ecae91d8fd1deedf944.js"></script></head><body><noscript><p>Your browser does not support JavaScript. Please enable JavaScript to view the site. Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.</p></noscript><div id="app"></div></body><script defer="defer" src="extensions.js"></script></html>
```

**Note:** As we use a self-sign certificate the `argocd-cmd-params-cm` ConfigMap in the argocd namespace should include the following setting `server.insecure: “true”`.

As expected, Sveltos took care of the complete lifecycle of the different Kubernetes deployments in a simple and straightforward manner! The example might seem simplistic but think about it on a bigger scale with hundreds of clusters across a multicloud setup. Imagine the flexibility of setting Sveltos labels to a cluster, defining the event source to trigger an action and having a Kubernetes deployment on a cluster. In the next post, we will demonstrate the power of Events in a cross multi-cluster environment.

👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos [GitHub repo](Projectsveltos GitHub repo). You can [star 🌟 the project](https://github.com/projectsveltos/addon-controller) if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!