BGP and Cilium Kubernetes
Tags: k8s, networking, firewall
Published: 2022-12-02 19:04:31

---
title: BGP and Cilium Kubernetes
published: true
date: 2022-09-20 05:00:00 UTC
tags: k8s,networking,firewall
canonical_url: https://jmoore53.com/k8s/networking/firewall/2022/09/20/bgp-cilium.html
---

For the current setup I have with kubernetes I ended up using Cilium as the networking provider configured with BGP on the OPNSense Firewall I have. This setup was a bit different because Cilium didn’t support having Nodes on the same network as the pods.. which makes sense, however it should have allowed for better configuration of the pod CIDR. Below is how I configured Cilium and OPNSense:

I made an attempt to run clium configured with BGP, but needed to move the nodes to 172.16.0.0/24 network.

This was the process I used:

- Create new vlan on switch for 172.16.0.0/12 network; 
  - Provisioning of nodes was done with Terraform;
- Pluck off a space in the network for kubernetes;
- Add Firewall interface for the new L2 network
- Update Terraform to move everything to a 172.16.0.0/12 network;
- Update Scripts
- Try Cilium again;

## Installing Helm and Cilium

```
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium --version 1.12.2 --namespace kube-system --set bgp.enabled=true --set bgp.announce.loadbalancerIP=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true #--set bgp.announce.podCIDR=true

```

After installing Clium I needed to add a config map for bgp pointing to opnsense (note this may have needed to have been done before installing cilium):if email_message.is_multipart():
    for part in email_message.get_payload():
        body = part.get_payload()
        # more processing?
else:
    body = email_message.get_payload()

- Note the peer address is my firewall ip
- Note the peer asn is the AS of the firewall
- Note the current AS is the AS of the kubernetes cluster

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: bgp-config
  namespace: kube-system
data:
  config.yaml: |
    peers:
      - peer-address: 172.16.0.254
        peer-asn: 64512
        my-asn: 64513
    address-pools:
      - name: default
        protocol: bgp
        addresses:
          - 172.16.1.0/24

```

## OPNSense Side:

On the opnsense side I had to install the `frr` package which allowed for better routing. From there I had to configure BGP.

This meant:

- enabling bgp
- Setting the BGP AS Number to 64512
- Adding all the k8s nodes into the neigbors of the bgp config with the correct peer asn (peer asns are 64513)

After everything has been configured LoadBalancer IPs can now be used on the 172.16.1.0/24 network.

## Links

- [BGP Cilium 1.12.2](https://docs.cilium.io/en/v1.12/gettingstarted/bgp/)
- 

| [BGP.html](http://docs.frrouting.org/en/stable-7.4/bgp.html#clicmd-[no]neighborPEERupdate-source%3CIFNAME | ADDRESS%3E) |

- [Cilium in VMs](https://help.accuknox.com/open-source/cilium-vm-k8s/)
- 

| [Helm | Installing Helm](https://helm.sh/docs/intro/install/) |

- [Service Map and Hubble UI](https://docs.cilium.io/en/v1.12/gettingstarted/hubble/#hubble-ui)
- [Using BGP to Integrate Cilium with OPNSense](https://dickingwithdocker.com/2021/05/using-bgp-to-integrate-cilium-with-opnsense/)