Multi-Cluster com Karmada [Lab Session]
Tags: kubernetes, karmada, community, aws
Published: 2024-05-15 00:46:05


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c8j8oubyo973zn9r252m.png)
> **[What is Karmada?](https://karmada.io/)**

> Karmada (Kubernetes Armada) is a Kubernetes management system that enables you to run your cloud-native applications across multiple Kubernetes clusters and clouds, with no changes to your applications. By speaking Kubernetes-native APIs and providing advanced scheduling capabilities, Karmada enables truly open, multi-cloud Kubernetes.

> Karmada aims to provide turnkey automation for multi-cluster application management in multi-cloud and hybrid cloud scenarios, with key features such as centralized multi-cloud management, high availability, failure recovery, and traffic scheduling.

> **[Architecture](https://karmada.io/docs/core-concepts/architecture)**
> ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f972taosi89ipyo1f3ze.png)
> _Source: https://karmada.io_

---

## Arquitetura do Lab

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tsyfmko6rbp1ss3s5ol6.png)

## Deploy da infra

- Cluster 1 ou _pegasus_:

```sh
git clone https://github.com/paulofponciano/EKS-Istio-Karpenter-ArgoCD.git
```

> [!NOTE]
> Altere os valores em _variables.tfvars_ se for necessário.
> No arquivo [nlb.tf](https://github.com/paulofponciano/EKS-Istio-Karpenter-ArgoCD/blob/main/nlb.tf), estamos informando um certificado do ACM. Altere para o seu certificado ou remova comentando as linhas 38, 39, 40 e descomentando a linha 37.

> Com certificado:
```tf
resource "aws_lb_listener" "ingress_443" {
  load_balancer_arn = aws_lb.istio_ingress.arn
  port              = "443"
  #protocol          = "TCP"
  protocol        = "TLS"
  certificate_arn = "arn:aws:acm:us-east-2:ACCOUNTID:certificate/bfbfe3ce-d347-4c42-8986-f45e95e04ca1"
  alpn_policy     = "HTTP2Preferred"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.https.arn
  }
}
```

> Sem certificado:

```tf
resource "aws_lb_listener" "ingress_443" {
  load_balancer_arn = aws_lb.istio_ingress.arn
  port              = "443"
  protocol          = "TCP"
  # protocol        = "TLS"
  # certificate_arn = "arn:aws:acm:us-east-2:ACCOUNTID:certificate/bfbfe3ce-d347-4c42-8986-f45e95e04ca1"
  # alpn_policy     = "HTTP2Preferred"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.https.arn
  }
}
```

```sh
tofu init
```
```sh
tofu plan --var-file variables.tfvars
```
```sh
tofu apply --var-file variables.tfvars
```

- Cluster 2 ou _pegasus-2_:

```sh
git clone https://github.com/paulofponciano/EKS-Istio-Karpenter.git
```

> [!NOTE]
> Altere os valores em _variables.tfvars_ se for necessário.

```sh
tofu init
```
```sh
tofu plan --var-file variables.tfvars
```
```sh
tofu apply --var-file variables.tfvars
```

## Deploy do karmada

```sh
git clone https://github.com/paulofponciano/karmada.git
```
```sh
cd karmada
```

Entrando no contexto do cluster _pegasus_:

```sh
aws eks update-kubeconfig --region us-east-2 --name pegasus
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9fqbzzyw39ua7fie3504.png)

> [!NOTE]
> Os comandos a seguir serão executados no cluster _pegasus_ (Cluster 1), onde temos o Argo rodando.

```sh
helm repo add karmada-charts https://raw.githubusercontent.com/karmada-io/karmada/master/charts
```
```sh
helm repo update
```
```sh
helm --namespace karmada-system upgrade -i karmada karmada-charts/karmada --create-namespace
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/df1i1zkll5ig98xhbsl2.png)

Verificando o deployment:

```sh
kubectl get pods -n karmada-system
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v9a5cb9cesiyxsawkqdi.png)

Agora já temos o controlplane do karmada rodando no cluster _pegasus_.

Com o deploy, é gerado um secret com o _kubeconfig_ necessário para conectarmos no controlplane do karmada:

```sh
kubectl get secrets -n karmada-system | grep karmada-kubeconfig
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lrgcy6muf21omavjwpyp.png)

```sh
kubectl get secret -n karmada-system karmada-kubeconfig -o jsonpath='{.data.kubeconfig}' | base64 --decode
```

## Criando IRSA (Iam Role for Service Account)

> [!NOTE]
> Altere os valores de _ACCOUNTID_ na _iam-policy-irsa-karmada.json_ e no comando _eksctl_ abaixo.

Isso refletirá em uma _Service Account_ dentro do cluster _pegasus_, usaremos ela montando em um pod do ubuntu como ponto de acesso **temporário** ao controlplane do karmada.

```sh
aws iam create-policy --policy-name ubuntu-admin-karmada \
  --policy-document file://iam-policy-irsa-karmada.json
```
```sh
eksctl create iamserviceaccount --name ubuntu-admin-karmada \
  --namespace karmada-system \
  --cluster pegasus \
  --attach-policy-arn arn:aws:iam::ACCOUNTID:policy/ubuntu-admin-karmada \
  --region us-east-2 \
  --profile default \
  --approve
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6n78zzv1zwxyax9gdlui.png)

No lado AWS, isso reflete em uma IAM role que podemos adicionar nos dois clusters EKS.

- EKS IAM - Console (pegasus-2):

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9rzy9egho6nfcutdztig.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pjcjgohxvi5zmscqut28.png)

Faça o mesmo para o cluster _pegasus_, onde o karmada controlplane está rodando.

## Acessando Karmada API-Server

Vamos subir agora aquele pod do ubuntu-admin no cluster _pegasus_. No manifesto já está tudo definido para utilizar a _Service Account_ que criamos mais acima.

```sh
kubectl apply -f https://raw.githubusercontent.com/paulofponciano/karmada/main/ubuntu-admin-karmada.yaml
```
```sh
kubectl get pods -n karmada-system | grep ubuntu
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wi2kplggwokyvd5i2cz6.png)

Nesse momento, vamos entrar no container do ubuntu, que está rodando no cluster _pegasus_:

```sh
kubectl exec -it ubuntu-admin-karmada -n karmada-system -- /bin/bash
```

- Instalando o kubectl karmada:

```sh
curl -s https://raw.githubusercontent.com/karmada-io/karmada/master/hack/install-cli.sh | bash -s kubectl-karmada
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jsb0pp7o2wcjkjrnw9ln.png)

- Entrando no contexto do cluster _pegasus-2_:

```sh
aws eks update-kubeconfig --region us-west-2 --name pegasus-2 --kubeconfig $HOME/.kube/pegasus-2.config 
```
```sh
kubectl get nodes --kubeconfig $HOME/.kube/pegasus-2.config
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9z9mye54042pukxyguvb.png)

- Vamos fazer o mesmo para o cluster _pegasus_:

```sh
aws eks update-kubeconfig --region us-east-2 --name pegasus --kubeconfig $HOME/.kube/pegasus.config
```
```sh
kubectl get nodes --kubeconfig $HOME/.kube/pegasus.config
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zgnpd144mdiawz9pu2fp.png)

- Checando acesso ao karmada apiserver:

```sh
kubectl get all -A --kubeconfig /etc/karmada-kubeconfig/kubeconfig
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lqwam5gsdpyjl13u2xvr.png)

- Join do _pegasus-2_ no karmada:

```sh
kubectl karmada --kubeconfig /etc/karmada-kubeconfig/kubeconfig join pegasus-2 --cluster-kubeconfig=$HOME/.kube/pegasus-2.config
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ymn4hlsauj5s88ytx5wk.png)

- Join do _pegasus_ no karmada:

```sh
kubectl karmada --kubeconfig /etc/karmada-kubeconfig/kubeconfig join pegasus --cluster-kubeconfig=$HOME/.kube/pegasus.config
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rz1gmoe0glvbjf373ko5.png)

- Checando status dos clusters adicionados:

```sh
kubectl --kubeconfig /etc/karmada-kubeconfig/kubeconfig get clusters
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ufvxbgr3hgw6jaox92gk.png)

- Instalando CLI do ArgoCD:

```sh
curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64
```

- Recuperando o secret / password do Argo server e fazendo login:

```sh
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
```

```sh
argocd login argocd-server.argocd.svc.cluster.local:80 --username admin
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6rixf69ushxkkd9lms7n.png)

- Adicionando o karmada como cluster no ArgoCD:

```sh
argocd cluster add karmada-apiserver --kubeconfig /etc/karmada-kubeconfig/kubeconfig --name karmada-controlplane
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/09h59u7ehw0wsqewlbge.png)

## Deploy com ArgoCD

Se acessarmos a UI do Argo, que está rodando no cluster _pegasus_, veremos que o karmada está registrado como um cluster onde é possível o Argo fazer deploy:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7b1kmg6nk86dngmtwr4e.png)

Podemos agora aplicar um manifesto que irá definir uma nova fonte para o Argo buscar por deploys. Nesse caso, essa fonte é um [repositório](https://github.com/paulofponciano/karmada/tree/main/app-manifests) no GitHub:

```sh
kubectl apply -f karmada-argo-app.yaml
```

Como já temos manifestos nesse repositório (no path **/app-manifests**), o Argo já faz o _sync_ entregando essas aplicações no controlplane do karmada e o karmada por sua vez, entrega nos dois clusters de acordo com o que for definido nos manifestos de _PropagationPolicy_:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/owmemh4zkjim1y1ztlez.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u0i9x0xbhxgq3sxlicme.png)

No cluster _pegasus_ podemos ver:

```sh
kubectl get pods -o wide | grep redis
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jtdtb2y6ivmj2wvo0fxl.png)

Cluster _pegasus-2_: 

```sh
kubectl get pods -o wide | grep nginx
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f8dxahyee76l6f3ub97k.png)

No caso do deploy do RabbitMQ, podemos ver que existem replicas rodando nos dois clusters, quando olharmos os aquivos de _PropagationPolicy_ poderemos entender.

```sh
kubectl get pods -o wide --context arn:aws:eks:us-east-2:ACCOUNTID:cluster/pegasus | grep rabbitmq
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mhrfn0ms7qba7kg34bfs.png)

```sh
kubectl get pods -o wide --context arn:aws:eks:us-west-2:ACCOUNTID:cluster/pegasus-2 | grep rabbitmq
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ywhd55lsm3042x7xcci.png)

## Karmada OverridePolicy e PropagationPolicy

No [repositório](https://github.com/paulofponciano/karmada/tree/main/app-manifests) que o Argo está monitorando, podemos ver os manifestos do karmada e também o manifestos de deployment que usamos como exemplo.

### Exemplo Redis

Regras de override e selector do deployment onde será aplicado, no caso 'redis':

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: redis-op
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: redis
  overrideRules:
    - targetCluster:
        clusterNames:
          - pegasus-2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              env: skoala-dev
          - operator: add
            value:
              env-stat: skoala-stage
          - operator: remove
            value:
              for: for
          - operator: replace
            value:
              bar: test
    - targetCluster:
        clusterNames:
          - pegasus
      overriders:
        annotationsOverrider:
          - operator: add
            value:
              env: skoala-stage
          - operator: remove
            value:
              bom: bom
          - operator: replace
            value:
              emma: sophia
```

Regras de propagação, selector do deployment e target cluster. Nesse caso de failover, esse deployment deve ser migrado para o cluster pegasus-2 caso o cluster pegasus entre em falha:

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: redis-propagation
spec:
  propagateDeps: true
  failover:
    application:
      decisionConditions:
        tolerationSeconds: 120
      purgeMode: Never
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: redis
  placement:
    clusterAffinity:
      clusterNames:
        - pegasus
        - pegasus-2
    spreadConstraints:
      - maxGroups: 1
        minGroups: 1
        spreadByField: cluster
```

### Exemplo Nginx

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: nginx-op
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  overrideRules:
    - targetCluster:
        clusterNames:
          - pegasus-2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              env: skoala-dev
          - operator: add
            value:
              env-stat: skoala-stage
          - operator: remove
            value:
              for: for
          - operator: replace
            value:
              bar: test
```

Neste caso, apenas o cluster _pegasus-2_ foi definido em 'targetCluster':

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - pegasus-2
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - pegasus-2
            weight: 1
```

### Exemplo RabbitMQ

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: rabbitmq-op
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: rabbitmq
  overrideRules:
    - targetCluster:
        clusterNames:
          - pegasus-2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              env: skoala-dev
          - operator: add
            value:
              env-stat: skoala-stage
          - operator: remove
            value:
              for: for
          - operator: replace
            value:
              bar: test
    - targetCluster:
        clusterNames:
          - pegasus
      overriders:
        annotationsOverrider:
          - operator: add
            value:
              env: skoala-stage
          - operator: remove
            value:
              bom: bom
          - operator: replace
            value:
              emma: sophia
```

Aqui temos algo diferente, onde os dois clusters são definidos em 'targetCluster', porém com pesos (weights) diferentes, fazendo com que o karmada entregue as réplicas de acordo com o peso de cada cluster:

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: rabbitmq-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: rabbitmq
  placement:
    clusterAffinity:
      clusterNames:
        - pegasus
        - pegasus-2
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - pegasus
            weight: 2
          - targetCluster:
              clusterNames:
                - pegasus-2
            weight: 1
```

---

### Remover ubuntu-admin e IRSA

Podemos deletar o pod ubuntu que usamos para setup do karmada, e também a IRSA:

```sh
kubectl delete -f ubuntu-admin-karmada.yaml
```
```sh
eksctl delete iamserviceaccount --name ubuntu-admin-karmada \                         
  --namespace karmada-system \
  --cluster pegasus \
  --region us-east-2 \
  --profile default
```

---

Para a próxima, vamos buscar um cenário total de DR com o karmada e ver até onde chegamos.

Keep shipping!