EKS and NetworkPolicies: the story so far
Published: 2023-09-12 00:08:43

On my monthly "let´s keep up with AWS news" live stream, one of the news caught my eye as **game changer**, and it was this one:

* [Amazon VPC CNI now supports Kubernetes NetworkPolicy enforcement](https://aws.amazon.com/about-aws/whats-new/2023/08/amazon-vpc-cni-kubernetes-networkpolicy-enforcement/)~

Let's have a peek on what this is about and why it is a game changer.

## Kubernetes NetworkPolicies

[NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) are often overlooked, but it is basically the major security feature of Kubernetes that grants you the hability to control the conectivity between your applications (and also the rest of the world!).

Many people get the feeling that the concept of **namespaces** provides isolation for applications running under, but it **does not provide network isolation by default**.

You can create a namespace **app1** and grant access for your **Team A** to deploy their applications. But if you allow **Team B** to run **jobs** on a namespace **app2**, even if they will not be able to modify app1 deployments, their jobs will be able to connect to the app1 deployments:

```bash
# Checking IPs for the applications:
$ kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.status.podIP}{"\n"}{end}' |
fgrep app
app1/backend-84bf889f7f-nrxbr: 192.168.130.72
app1/frontend-54d8796d8c-2fzsk: 192.168.140.197
app2/job-f8dd4484b-kmxkm: 192.168.138.201

$ for i in 192.168.130.72 192.168.140.197; do \
    kubectl -n app2 exec job-f8dd4484b-kmxkm -- curl $i -so /dev/null -w '%{HTTP_CODE}' \
  done
200
200
```

This may not sound like a problem to you, but it is certainly not the most desirable situation, specially if most of your security is invested into the LoadBalancers/Web Application Firewalls that are outside the security perimeter of your cluster - it turns your cluster into a lateral movement extravaganza if someone not intended manages to get access to it.

## Securing the application 

In order to block traffic from other namespaces to your application, you have to apply a **NetworkPolicy** that denies all incoming traffic:

```bash
$ echo '---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: app1
spec:
  podSelector: {}
  policyTypes:
  - Ingress
' | kubectl apply -f -
```

Syntax might look weird at a glance, but it applies to all pods (podSelector: {}) allowing **nothing** because there is no **ingress** block defined at the top level of spec.

After these rules get applied, no one will be able to access pods on namespace **app1**:

```bash
$ for i in 192.168.130.72 192.168.140.197; do
    kubectl -n app2 exec job-f8dd4484b-kmxkm -- curl --connect-timeout 2 $i -so /dev/null -w '%{HTTP_CODE}\n'
  done
000
command terminated with exit code 28
000
command terminated with exit code 28
```  

Problem is **not even applications on app1 will be able to access each other**. Once you apply any networkpolicy of a type (ingress or egress), all traffic that is not specificly allowed will be **denied**.

So if you have a frontend/backend combo like below:
```
$ kubectl -n app1 get pods -o name
pod/backend-84bf889f7f-nrxbr
pod/frontend-54d8796d8c-2fzsk
```

You should allow incoming conections to backend only from the **frontend** deployment, and frontend might get its connections from a **Load Balancer** outside the cluster.

To allow frontend to access backend, one would leverage the **kubernetes labels** to make the rule **dynamic**:

```bash
$ kubectl -n app1 get pods --show-labels | tr -s ' ' | cut -f1,6 -d' '
NAME LABELS
backend-84bf889f7f-nrxbr app=backend,pod-template-hash=84bf889f7f
frontend-54d8796d8c-2fzsk app=frontend,pod-template-hash=54d8796d8c
```

**Backend** pods will always have the label **app=backend**, and **Frontend** pods will have the label **app=frontend**; this will tell the NetworkPolicy Controller to update the rules every time a new pod is created (or destroyed).

```bash
# Conecting to the **service** backend from the pod frontend
$ kubectl -n app1 exec deploy/frontend -- curl --connect-timeout 2 -so /dev/null http://backend  -w '%{http_code}'
000
command terminated with exit code 28

$ echo '---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-to-backend
  namespace: app1 
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 80
' | kubectl apply -f -

# Retesting:
$  kubectl -n app1 exec deploy/frontend -- curl --connect-timeout 2 -so /dev/null http://backend  -w '%{http_code}'
200
```  

Unfortunately, there is no easy way to restrict access to the frontend pods from the load balancers, because they're not hosted inside the cluster. 

(You can use [Security Groups for Pods](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) for that!)

The best you can do is to limit connections from the subnets where the load balancers will create their ENIs (or use their ips, if you feel bold!):

```bash
$ echo '---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: lb-to-frontend
  namespace: app1
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 192.168.0.0/19
        - ipBlock:
            cidr: 192.168.32.0/19
        - ipBlock:
            cidr: 192.168.64.0/19
      ports:
        - protocol: TCP
          port: 80
' | kubectl apply -f - 
```  

Of course, anything else that is created on those subnets will be able to access the frontend pods on namespace app1.

The backend pod, otherwise, won't be able to start any connection to the frontend:

```bash
$ k -n app1 exec -it deploy/backend -- curl frontend:80 -so /dev/null -w '%{http_code}' --connect-timeout 2
000
```

Kubernetes NetworkPolicies allow an incredible degree of microsegmentation with little to no effort, even allowing for the devs themselves to be responsible to translate their integrations in a declarative way.

But of course, there is always a catch.

## NetworkPolicy Controller

Unfortunately, Netpols are one of the few native Kubernetes Resources that **do not have a default controller assigned to it** - the other major one would be [Ingresses](https://kubernetes.io/docs/concepts/services-networking/ingress/).

Even though it's clearly stated in the docs, as quoted bellow:

> Network policies **are implemented by the network plugin**. To use network policies, **you must be using a networking solution which supports NetworkPolicy**. Creating a NetworkPolicy resource **without a controller** that implements it **will have no effect**.
> [source](https://kubernetes.io/docs/concepts/services-networking/network-policies/#prerequisites)

It's still easy for people to miss it, specially if they are less kubernetes-savy than they should be when using this type of technology.

And yes, I've been asked "why my netpols are not working" before. People just assume that it should work.

So, if you install an EKS cluster right now and reproduce the same configurations I have listed here, **nothing will work**.

At least for a while, because things changed!

## EKS and NetworkPolicies

Until August 2023, the only way use NetworkPolicies was to deploy a third party software called [Project Calico](https://www.tigera.io/project-calico/). It's a full fledged CNI, but one would enable only the Policy part as [described in the official EKS docs](https://docs.aws.amazon.com/eks/latest/userguide/calico.html).

But now, things changed! If you install a new EKS Cluster with the CNI version 1.14.0 or above, [it now supports NetworkPolicies natively!](https://aws.amazon.com/about-aws/whats-new/2023/08/amazon-vpc-cni-kubernetes-networkpolicy-enforcement/)!

All you have to do is to create your EKS Cluster with the following on your eksctl yaml:

```bash
...
addons:
- name: vpc-cni 
  version: 1.14.0
  configurationValues: |-
    enableNetworkPolicy: "true"    
  attachPolicyARNs:
  - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
...
```  

And that's it! Awesome news for EKS users!

## Final notes

I believe it's worth noting that AWS CNI native NetworkPolicies make use of [**eBPF**](https://ebpf.io/) and not **iptables** as many of the other solutions available.

It does not support port translation from Services:

> For any of your Kubernetes services, the service port must be the same as the container port. If you're using named ports, use the same name in the service spec too.

For a brief period during creation, the pod will not have any restrictions applied to it: 

> The Amazon VPC CNI plugin for Kubernetes configures network policies for pods in parallel with the pod provisioning. Until all of the policies are configured for the new pod, containers in the new pod will start with a default allow policy. All ingress and egress traffic is allowed to and from the new pods unless they are resolved against the existing policies.

It's also not supported on Fargate or Windows nodes. 

## Links

* https://aws.amazon.com/about-aws/whats-new/2023/08/amazon-vpc-cni-kubernetes-networkpolicy-enforcement/
* https://aws.amazon.com/blogs/containers/amazon-vpc-cni-now-supports-kubernetes-network-policies/