Costly EKS Cluster Set Up? AWS to the rescue!
Tags: beginners, aws, kubernetes, eks
Published: 2023-04-23 18:10:03

## Storyline
The goal was to do an automated **Terraform Kubernetes Deployment**. I eventually provisioned a production grade EKS Cluster using Terraform-EKS-modules and it created other AWS resources to be intergrated along with it. As the beginner that I was, I left the cluster running for days hoping to sort out deploying different types of microservice applications to the same cluster then finally clean up. The total bill accrued after a while was jaw-dropping and problematic but **AWS** came to my rescue and sorted it all out.  

## Aim
This walks you through the process of creating a standard EKS-Cluster using Terraform modules in [AWS EKS Cluster Creation](#aws-eks-cluster-creation) and steps to request a total cost/bills waiver from AWS in [AWS Bills Waiver](#aws-bills-waiver). The waiver process is useful for other workloads and not just this cluster set up.

## [AWS EKS Cluster Creation](#aws-eks-cluster-creation)
### Essentials:
- An AWS Account, create a free tier [here](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html?refid=99f831a2-d162-429a-9a77-a89f6b3bd6cd).
- Be logged in an IAM user with Admin priviledges, secret and access keys. 
- Ubuntu Host OS (Local PC or Virtual Machine).
- VS code or any IDE of your choice.
- An S3 bucket to store the terraform state files remotely
- Basic Linux Terminal use

### Installations:
- Install latest **Terraform** on the command line:
```shell
sudo apt update && apt upgrade -y
sudo apt install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update
sudo apt install terraform
terraform -v
```
![Terraform's version Output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rxt3fa0wsh85p94lq7j5.png)
- Install the **Terraform extension** by _Anton Kulikov_ on VS code for syntax highlights and efficiency.
- Install **AWS CLI** :
```shell
sudo apt-get update
sudo apt-get install -y less curl unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -q awscliv2.zip
sudo ./aws/install
aws --version
```
![AWS CLI's version Output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sxmufzqsxq6rwt7imsu2.png)

- Configure AWS CLI with `aws configure` command:
    - fill up outputs with secret and access keys previously created from IAM user
	- add `us-east-1` as region 
	- Test configuration details with `aws configure list`
	- No need to create environment variables as runnng `env | grep AWS` will not recognize the enviroment variables set in another terminal tab hence `aws configure` is appropriate for global setup.
    ```shell
    export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
    export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
    export AWS_DEFAULT_REGION="us-east-1"
    ```
<br>

- Install **Kubectl** as the command line tool that interacts with the Kubernetes Cluster:
```shell
sudo apt-get update -y
sudo apt-get install -y apt-transport-https ca-certificates curl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -y
sudo apt-get install -y kubectl
kubectl version --client
```
<br>

- (Optiona) Install **Graphviz** to visualize the terraform plan process:
```shell
sudo apt install graphviz
```

### Architecture
![Set Up](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0xkthnfad2vkudpmrtyo.png)
This shows the summarized version of how the cluster would be formed via the terraform module(more like a group of interwoven resources used together). Other resources such as VPC, Security groups, NAT and Internet gateways for routing traffic from inside the private subnets and the internet respectively..etc. would be created alongside it. 
This set up has the VPC created with three Public and three Private Subnets which tallies with the three availabilty zones present in `us-east-1` region.
<br>
The values in the Terraform Configuration aren't fixed and can be changed according to individual's preference but I'll suggest that beginners follow through these exact values in order to eradicate bugs and avoidable errors.

### Procedures:
- Navigate to preferred working directory by git cloning my repository 
`git clone https://github.com/wandexdev/k8s-microservices-deployment-via-CICD-terraform.git`, then `cd terraform/eks_creation-terraform/` to switch directory.
<br>

> **Note**
>
> Ensure your present working directory is `~/k8s-microservices-deployment-via-CICD-terraform/terraform/eks_creation-terraform`
>
> Confirm with the command `pwd`.

- All required Terraform files(.tf) are present in the folder on my repo, you can create a folder for yours to manually write the `.tf` files if you **did not** clone the repository.
<figure>
    <img
    src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zfktp324wjvi214lackd.png"
    alt="Terraform">
    <figcaption>All required .tf files</figcaption>
</figure>

- The majority configuration of the module used was gotten from [terraform-aws-modules/eks in Terraform's Registry](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest?tab=inputs). You can add more/switch the **non required** inputs or decide which preferred resources or versions you want outside the ones I already chose in my repository.

- Create a **providers.tf** file to recognize the specific terraform-provider plugins that directly interact with AWS service.

```

        provider "aws" {
            region  = var.region
        }
```
<br>

- Create a **s3_backend.tf** file to store remote state files in an external S3 bucket as this ensures multiple users or CI server gets the latest state data before running terraform. Its also more secure.
> **PS**
>
> Please ensure the S3 bucket is created prior to terraform being initialized
<br>

```
        
        terraform {
        required_version = ">= 0.12"
        backend "s3" {
            bucket         = var.S3_bucket_name
            key            = var.bucket_key
            region         = var.region
        }
     }
```
<br>

- Create a **variables.tf** file to declare all variables to be used in this project.

```
        
        #----------------------------BACKEND-------------------------
        variable "S3_bucket_name" {
            description = "name of S3 bucket for remote backend storage"
        }

        variable "bucket_key" {
            description = "It's the file path inside the bucket"
        }

        variable "region" {
            description = "The region of cluster deployment"
        }

        #--------------------------------VPC-------------------------------#

        variable "vpc_cidr_block" {
            description = "ip range for vpc"
        }

        variable "private_subnet_cidr_blocks" {
            description = "ip range for private subnet"
        }

        variable "public_subnet_cidr_blocks" {
            description = "ip range for public subnets"
        }
```

<br>

- Create a **vpc.tf** file for the configurations of the Virtual Private Environment the EKS-cluster would be created in. 

```

        #-----------DATA SOURCE-------------------#
        # By state
        data "aws_availability_zones" "available" {
        }

        module "wandek8s-vpc" {
        source                                    = "terraform-aws-modules/vpc/aws"
        version                                   = "2.64.0"

        name                                      = "wandek8s-vpc"
        cidr                                      = var.vpc_cidr_block
        private_subnets                           = var.private_subnet_cidr_blocks
        public_subnets                            = var.public_subnet_cidr_blocks
        azs                                       = data.aws_availability_zones.available.names 
            
        enable_nat_gateway                        = true
        single_nat_gateway                        = true
        enable_dns_hostnames                      = true

        tags                                      = {
                "kubernetes.io/cluster/wandek8s-eks-cluster" = "shared"
        }

        public_subnet_tags                        = {
                "kubernetes.io/cluster/wandek8s-eks-cluster" = "shared"
                "kubernetes.io/role/elb"            = 1
        }

        private_subnet_tags                       = {
                "kubernetes.io/cluster/wandek8s-eks-cluster" = "shared"
                "kubernetes.io/role/internal-elb"   = 1
        }
        }
```
<br>

- Create an **outputs.tf** file to share data between Terraform configurations and external tools. Outputs are also the only way to share data between child module and root module of the configuration.

```

    output "cluster_id" {
    description       = "EKS cluster ID."
    value             = module.eks.cluster_id
    }

    output "cluster_name" {
    description       = "Amazon Web Service EKS Cluster Name"
    value             = module.eks.cluster_name
    }

    output "cluster_endpoint" {
    description       = "Endpoint for Amazon Web Service EKS "
    value             = module.eks.cluster_endpoint
    }

    output "region" {
    description       = "Amazon Web Service EKS Cluster region"
    value             = var.region
    }

    output "cluster_security_group_id" {
    description       = "Security group ID for the Amazon Web Service EKS Cluster "
    value             = module.eks.cluster_security_group_id
    }

    output "cluster_ca_certificate" {
    value             = module.eks.cluster_certificate_authority_data
    }
```
<br>

- Create an **eks-cluster.tf** file which specifies the Terraform Kubernetes Provider in order to create Kubernetes objects, the module configurations, the partly managed nodes groups + node instances to be created in the cluster etc.

```

        #-----------provider to communicate with Kubernetes’ resources----------#

        provider "kubernetes" {
            host                        = module.eks.cluster_endpoint # end point of k8s (API server)
            token                       = data.aws_eks_cluster_auth.wandek8s-eks-cluster.token
            cluster_ca_certificate      = base64decode(module.eks.cluster_certificate_authority_data)
        }

        #--------Query data ------------#
        #data "aws_eks_cluster" "wandek8s-eks-cluster" {
        #    name                        = module.eks.cluster_id
        #}

        data "aws_eks_cluster_auth" "wandek8s-eks-cluster" {
            name                        = module.eks.cluster_name
        }

        module "eks" {
        source                        = "terraform-aws-modules/eks/aws"
        version                       = "19.10.0"

        cluster_name                  = "wandek8s-eks-cluster"
        cluster_version               = "1.24"

        subnet_ids                    = module.wandek8s-vpc.private_subnets
        vpc_id                        = module.wandek8s-vpc.vpc_id
        cluster_endpoint_public_access = true

        tags = {
            environment = "development"
            application = "wandek8s"
        }

        eks_managed_node_groups = {
            one = {
            min_size     = 1
            max_size     = 4
            desired_size = 3

            instance_types = var.instance_type
            }
        }
        }
```
<br>

- Finally Create a **terraform.tfvars** file to input the variable values defined in **variables.tf** file. Herein lies the ranges for VPC CIDR block, specific resources names etc. Please ensure to input the absolute path for private and public key located in the HOST OS as mine is removed for security reasons. 

> This is a very sensitive file that **shouldn't** be pushed to version control.

```

        S3_bucket_name               = "wandek8s-bucket"

        bucket_key                   = "cicd-server/state.tfstate"

        region                       = "us-east-1"

        key_pair                     = "cicd-server"

        local_public_key_location    = ""

        local_private_key_location   = ""

        instance_type                = ["t3.medium"]

        vpc_cidr_block               = "10.0.0.0/16"

        public_subnet_cidr_blocks    = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]

        private_subnet_cidr_blocks   = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
```

<br>

- Initialize the current working directory with `terraform init`
<figure>
    <img
    src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2beeu6q3s7tmad3fad07.png"
    alt="Terraform">
    <figcaption>Terraform init Output</figcaption>
</figure>
- Next `terraform plan` to preveiw the list of the desired state to be executed.
<figure>
    <img
    src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/durxl6ax9z55fr2nq59d.png"
    alt="Terraform">
    <figcaption>Terraform plan part Output</figcaption>
</figure>
- Optionally, we can visualize the terraform plan process before finally creating the resources in a graph via `terraform graph -type plan  | dot -Tsvg > graph.svg`. A graph.svg file would be created that can abe viewed on your PC or via any svg viewing tool.
- Next is `terraform apply` and patience because this process would take a little while.
<figure>
    <img
    src=""
    alt="Terraform">
    <figcaption>Terraform apply Output</figcaption>
</figure>
- Console Verification: After a successful apply, we can confirm the resources created on the AWS console
![console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0tsfw8rgmve6rf3o5wt0.png)
![console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g5ftu557wxkr24uimbh2.png)
![console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wmf6haeo71nkw49e5wm9.png)

- Validate the cluster with kubectl via `aws eks update-kubeconfig --name wandek8s-eks-cluster` and now the cluster is fully ready for interaction and deployments.

- Please do remember to **Clean up** the infrastructure as the spend for this set up is costly. Terraform makes this super easy by typing `terraform delete`. If not proceed to the next section where I explain how I got my bills waived as a 1 time occurence. 

## [AWS Bills Waiver](#aws-bills-waiver).
### Steps:
- Navigate to the AWS console and select support centre from the drop down list
<figure>
    <img
    src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hobimutbr09smx8wory7.png"
    alt="Waiver">
    <figcaption>Figure 1: Support Centre on web console</figcaption>
</figure>

- Ensure **Account & Billing** is ticked then proceed to **Create Case**
<figure>
    <img
    src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tujlqrje4bf2o3wtuuce.png"
    alt="Waiver">
    <figcaption>Figure 2: Create Case in Account & Billing</figcaption>
</figure>

- Follow the option specifications in Figure 3 below.
<figure>
    <img
    src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/njzwqpwu0bk0r3c56kfm.png"
    alt="Waiver">
    <figcaption>Figure 3: Selecting Service and Category options</figcaption>
</figure>

- Fill the **Subject** **Summary**. I recommend an honest breakdown of the events that lead to the cost build, documents can be attached also. Here is my description version here, you can use it as a template.
![Case summary](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z9mrjxvmafk8eruot62s.png)

```
Hello AWS Support team,

I am requesting your assistance with my recent AWS charges. As a beginner to AWS processes, I have incurred very high unexpected charges that I am not able to afford at this moment while setting up an EKS Cluster with modules and I am hoping to find a solution that will allow me to continue learning and using AWS services without experiencing any financial burden.


I understand that some of the charges may have been due to my lack of knowledge about AWS pricing and usage. I have since taken steps to educate myself and ensure that I am aware of the free tier limits and best practices for minimizing costs and curbing hidden costs.


I would be very grateful if my account be waived off all the pending AWS charges on my account as a one-time courtesy due to my status as a new user. This would allow me to continue my learning journey with AWS without any worry about unexpected charges.


Thank you for your time and attention to this matter. I appreciate your assistance in resolving this issue.
```

- Select **Web** and you'll be contacted via chat with a wave off alongside few aws credits to clear future debts. Depending on your location, you can also select **Phone**, provide your phone number and you'll be contacted by the team.
![Contact](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xcu4pw2hng3upn4qlywm.png)

<br>
Thank you for coming on this journey with me and I hope I have been able to guide you in 2 different ways with my experiences with AWS awesomeness. 

<br>
Feel free to reach out in the comment section with doubts or questions as well as directly on [twitter](https://twitter.com/Wandexdev). Cheers to a fruitful learning journey with AWS.