Ultimate EKS Baseline Cluster: Part 1 - Provision EKS
Tags: kubernetes, devops, awscloud, tutorial
Published: 2023-07-21 19:14:52

The AWS managed Kubernetes service, [EKS](https://aws.amazon.com/eks/) ([Elastic Kubernetes Service](https://aws.amazon.com/eks/)), has the highest level of complexity amongst cloud offerings. 

Besides having to build out the networking, routing, security, and worker nodes separately from the managed master nodes, there’s no longer any bundled support for storage starting from 1.23, and the existing support for external load balancers is limited.

This tutorial, presented in separate parts, will demonstrate how to build out a robust Kubernetes cluster with just a few commands. 

1. EKS Cluster with support for OIDC
2. Add Storage support: [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) using [EBS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html)
3. Add load balancer support: layer 4 ([NLB](https://aws.amazon.com/elasticloadbalancing/network-load-balancer/)) and layer 7 ([ALB](https://aws.amazon.com/elasticloadbalancing/application-load-balancer/))
5. Add traffic security support: network policies using [Calico](https://www.tigera.io/project-calico/)
6. Demo application with [Dgraph](https://dgraph.io/) putting it all together

In order to secure services that need access to cloud resources, an [OIDC provider](https://openid.net/developers/how-connect-works/) through [IAM Roles for Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) facility will be used.

## 0. Prerequisites

These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.

### 0.1 Knowledge: Systems

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aqzshuffp4kni7i9t487.png)

Basic concepts of systems, such as Linux and the shell (redirection, pipes, process substitution, command substitution, environment variables), as well as virtualization and containers are useful. The concept of a service (daemon) is important.

### 0.2 Knowledge: Networking

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5hd97akfkxxg8dzg26mh.png)

This article requires some basic understanding of networking with TCP/IP and the OSI model, specifically the Transport Layer 4 and Application Layer 7 for HTTP. This article covers using load balancing and reverse proxy.

In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.

Exposure to other types of Kubernetes resource objects used in this guide are helpful: `persistentvolumeclaims` (`pvc`), `storageclass` (`sc`), `pods`, `deployments`, `statefulsets` (`sts`), `configmaps`, `serviceaccount` (`sa`) and `networkpolicies`.

### 0.4 Tools

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7u87et62nyhbwrksb29d.png)

These are the tools used in this article series:

* [AWS CLI](https://aws.amazon.com/cli/) [`aws`] is a tool that interacts with AWS.
* [kubectl client](https://kubernetes.io/docs/reference/kubectl/) [`kubectl`] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
* [helm](https://helm.sh/) [`helm`] is a tool that can install Kubernetes applications that are packaged as helm charts.
* [eksctl](https://eksctl.io/) [`eksctl`] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
* [POSIX Shell](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html) [`sh`] such as [bash](https://www.gnu.org/software/bash/)[`bash`] or [zsh](https://www.zsh.org/) [`zsh`] are used to run the commands.

These tools are highly recommended:

* [adsf](https://asdf-vm.com/) [`adsf`] is a tool that installs versions of popular tools like kubectl.
* [jq](https://jqlang.github.io/jq/) [`jq`] is a tool to query and print JSON data
* [GNU Grep](https://www.gnu.org/software/grep/) [`grep`] supports extracting string patterns using extended [Regex](https://en.wikipedia.org/wiki/Regular_expression) and [PCRE](https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions).

### 0.5 AWS Setup

Before getting started on EKS, you will need to set up billing to an AWS account (there’s a free tier), and then configure a profile that has provides access to an IAM User identity. See Setting up the AWS CLI for more information on configuring a profile.

After setup, you can test access with the following below:

```bash
export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity
```

This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named `kwisatzhaderach`:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vqlvd4rpb6fj0763mwrj.png)

### 0.6 Kubernetes Client Setup

If you use `asdf` to install `kubectl`, you can get the latest version with the following:

```bash
# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null
```

This should show something like:

```yaml
Client Version: v1.27.1
Kustomize Version: v5.0.1
```

Also, create directory to store Kubernetes configurations that will be used by the `KUBECONFIG` env variable:

```bash
mkdir -p $HOME/.kube
```

### 0.7 Setup Environment Variables

These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.

```bash
# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"
# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)
```

## 1. Provision an EKS cluster

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fu5ewlzhqd5qdgjdvfg9.png)

After prerequisite tools are installed and setup, we can start provisioning cloud resources and deploy components to Kubernetes. 

### 1.1. Provision the EKS Cluster

The cluster can be brought up with the following command:

```bash
eksctl create cluster \
  --version $EKS_VERSION \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --nodes 3
```

Also, check the status of the worker nodes and applications running on Kubernetes.

```bash
kubectl get nodes
kubectl get all --all-namespaces
```

This should show something like the following.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hdlwkg7iospru75jpyki.png)

### 1.2. Install the Kubernetes client

Once this finished in about 20 minutes to provision both VPC and EKS, install a kubectl version that matches the Kubernetes server version:

```bash
# fetch exact version of Kubernetes server (Requires GNU Grep)
VER=$(kubectl version --short 2> /dev/null \
  | grep Server \
  | grep -oP '(\d{1,2}\.){2}\d{1,2}'
)

# setup kubectl tool
asdf install kubectl $VER
asdf global kubectl $VER
```

> ⚠️ **NOTE**: The above command requires [GNU grep](https://www.gnu.org/software/grep/). If you have [Homebrew](https://brew.sh/), you can run `brew install grep`. Windows can get this with [MSYS2](https://www.msys2.org/) or [git-bash](https://git-scm.com/download/win).

### 1.3 Add OIDC Provider Support

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gesyugb0faz6d5qxz8bf.png)

The EKS cluster has an [OpenID Connect](https://openid.net/developers/how-connect-works/) ([OIDC](https://openid.net/developers/how-connect-works/)) issuer URL associated with it. To use [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) ([IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)), an IAM OIDC provider must exist for the cluster’s OIDC issuer URL.

You can set this up with the following command:

```bash
eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve
```

You can verify the [OIDC provider](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) is added with the following:

```bash
OIDC_ID=$(aws eks describe-cluster \
  --name $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --query "cluster.identity.oidc.issuer" \
  --output text \
  | cut -d '/' -f 5
)

aws iam list-open-id-connect-providers \
  | grep $OIDC_ID \
  | cut -d '"' -f4 \
  | cut -d '/' -f4
```

## Resources

### AWS Cloud Resources

These are some of the AWS resource objects used in this guide.

* [Elastic Kubernetes Service](https://aws.amazon.com/eks/) ([EKS](https://aws.amazon.com/eks/))
* [Virtual Compute Cloud](https://aws.amazon.com/vpc/) ([VPC](https://aws.amazon.com/vpc/))
* [Elastic Computer Cloud](https://aws.amazon.com/ec2/) ([EC2](https://aws.amazon.com/ec2/))
* [AutoScale Group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html) ([AGS](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html))
* [Managed Node Groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)

## Conclusion

This first article shows have to get a modern Kubernetes cluster working with support for OIDC that will allow restricted access to cloud resources from Kubernetes applications using concept of [least privilege](https://wikipedia.org/wiki/Principle_of_least_privilege)

This article uses `eksctl` tool to provision the necessary cloud resources to bring up EKS.  The `eksctl` tool will use [Cloud Formation](https://aws.amazon.com/cloudformation/) stacks to deploy the necessary cloud resources (VPC, EC2, SGs, etc) that make up the full solution.

I am sure there are many wondering, why not use something like [Terraform](https://www.terraform.io/) for provisioning? 

That is actually a good idea except that this will be more involved because we'll have to deploy all of the components required to make a functional EKS cluster: VPC, internet gateways, nat groups, subnet, routing tables, auto-scale groups, security groups, etc. in addition to the master control plane (EKS).

In the future, I would like to make a similar series with Terraform to setup the base EKS cluster, and then use [Terraform](https://www.terraform.io/) with [templatefile](https://developer.hashicorp.com/terraform/language/functions/templatefile) or [Helmfile](https://tanzu.vmware.com/developer/guides/helmfile-what-is/) to manage installing helm charts and kubernetes manifests.

The ultimate goal of this series is to quickly get a robust and secure Kubernetes cluster with necessary features like storage and networking with preferably low complexity and friction.

From here, we can explore other developments and tutorials on Kubernetes, such as o11y or observability ([PLG](https://grafana.com/oss/loki/), [ELK](https://www.elastic.co/what-is/elk-stack), [ELF](https://fluentbit.io/), [TICK](https://www.influxdata.com/blog/introduction-to-influxdatas-influxdb-and-tick-stack/), [Jaeger](https://www.jaegertracing.io/), [Pyroscope](https://pyroscope.io/)), service mesh ([Linkerd](https://linkerd.io/), [Istio](https://istio.io/), [NSM](https://www.nginx.com/blog/introducing-nginx-service-mesh/), [Consul Connect](https://developer.hashicorp.com/consul/docs/connect), [Cillium](https://isovalent.com/blog/post/cilium-service-mesh/)), and progressive delivery ([ArgoCD](https://argo-cd.readthedocs.io/), [FluxCD](https://fluxcd.io/), [Spinnaker](https://spinnaker.io/)).