Provision EKS Cluster and apply Kubernetes configuration via CloudFormation
Tags: eks, cloudformation, kubernetes, lambda
Published: 2022-08-28 16:30:00

_CloudFormation_ is a great solution for provisioning AWS infrastructure. But when it comes to provisioning Amazon Elastic Kubernetes Service (EKS) Cluster using _CloudFormation_ and bootstrapping it to apply _Kubernetes_ configuration, then we face some interesting challenges.

In this article, we will discuss these challenges in more details. I will explain how my _CloudFormation_ backed serverless solution can help automate provisioning EKS resources (Cluster, Node Group) and apply _Kubernetes_ configuration automatically after the cluster is created.

This article was also published on [Medium](https://medium.com/@chandranathmondal/self-service-amazon-eks-cluster-provisioning-with-kubernetes-configuration-applied-372bce839d7).

{% embed https://medium.com/@chandranathmondal/self-service-amazon-eks-cluster-provisioning-with-kubernetes-configuration-applied-372bce839d7 %}

## Problem Statement

As per [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html):

> When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. To grant additional AWS users or roles the ability to interact with your cluster, you must edit the _aws-auth ConfigMap_ within _Kubernetes_ and create a _Kubernetes_ rolebinding or clusterrolebinding with the name of a group that you specify in the _aws-auth ConfigMap_.

That means, we can’t specify which users/groups to have access to a Cluster during provisioning it. It’s always initialised with the calling identity only. As a next step, we must apply _Kubernetes_ configuration to update the _aws-auth ConfigMap_. This is achieved by executing a _kubectl_ command.

Here comes our first challenge!

### CHALLENGE 1: How to execute kubectl command from CloudFormation?

We need to make _CloudFormation_ invoke the _kubectl_ command execution after the Cluster is created.

**CodeBuild**
It’s a build platform where we can install _kubectl_.

**Custom Resource**
We can deploy a _Lambda_ backed _CloudFormation Custom Resource_ just after the cluster is created. The _Custom Resource_ would make a _Lambda_ function call which can trigger a build in _CodeBuild_ project to initialise the ConfigMap.

But now we have a bigger challenge!

### CHALLENGE 2: Identity problem

In enterprise scenarios where infrastructure is provisioned via _CloudFormation_, the resource provisioning permission is assigned to the _CloudFormation Service Role_ so that end users don't need to have those permissions.

Hence, when we create an EKS Cluster via _CloudFormation_, EKS will initialise new Cluster’s RBAC so that only the _CloudFormation Service Role_ have access to it. As a next step, you have to execute a _kubectl_ command on _CodeBuild_ but at this point only the _CloudFormation Service Role_ can do that. Then, how would _CodeBuild_ assume the _CloudFormation Service Role_?

**Service Role with multiple service principals**
We can create a common service role with multiple service principals e.g. _CloudFormation_, _CodeBuild_ etc. In this case, the identity which creates the Cluster is the same identity that executes the _kubectl_ command on _CodeBuild_.

## Solution Overview

EKS-Factory automates provisioning of EKS resources (Cluster, Node Group) and configuring these resources in a self-service manner. As explained above, we use _CloudFormation_ to provision these resources. We also leverage _Lambda_ and _CodeBuild_ to configure the provisioned resources.

All the source code and configuration details, discussed here, are available at the [Github repository](https://github.com/chandranathmondal/eks-factory). Please refer to [README.md](https://github.com/chandranathmondal/eks-factory/blob/main/README.md) file for installation instructions.

## Solution Architecture

Following is the high level solution architecture diagram showing different components and underlying resources that get provisioned:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0x5hiqsfv4ixgd1elbym.png)

When we provision any EKS resources, we will provision it in a logical boundary, we call it as “environment”. There are two different types of stack that we provision for any EKS environment:

### STACK 1: EKS Control Plane/Environment

This will be the first stack created for an EKS environment and there will be only one _EKS Control Plane_ stack per environment. This stack deploys the actual EKS Cluster along with other resources. Following is the workflow of different resources being provisioned in this stack. A child/nested stack is created in each of these steps making it easy to maintain the underlying resources.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/54nkrufy5w08o5yd3p8x.png)

**STEP 1: Deploy Config Manager**
First, we deploy a _Lambda_ function, we call it as Config Manager as it performs all the configuration required at different stages of resource provisioning. Before that, we create the IAM Role (Config Manager Role) required for that _Lambda_ function. This _Lambda_ function allows us to provision _Lambda_ backed _Custom Resources_ in _CloudFormation_ which in-turn makes the _Lambda_ function call as and when required to perform required configuration changes.

**STEP 2: Deploy Config Map Manager**
Next, we create a _CodeBuild_ project, we call it as Config Map Manager as it configures the _aws-auth ConfigMap_ in _Kubernetes_ as and when required. To do that, we start a build in _CodeBuild_ project via the _Lambda_ function call. We store all our script files in S3 and point our _CodeBuild_ project to use the S3 location as repository.

> These two components enable us to perform all configuration management tasks as part of the resource provision.

**STEP 3: Provision Prerequisites**
Here, we create the Cluster Security Group and Node Security Group.

**STEP 4: Provision Cluster**
Then we create the _Cluster Service Role_ and provision the actual EKS Cluster.

**STEP 5: Deploy Add-ons**
Add-ons are individual resources in _CloudFormation_. Here we provision vpc-cni, kube-proxy and coredns add-ons.

**STEP 6: Enable Logging**
In this step we provision a _Lambda_ backed _Custom Resource_ which in turn makes a call to the Config Manager (_Lambda_ function) and enables API Server, Audit, Authentication, Controller Manager and Scheduler logging.

**STEP 7: Initialise Config Map**
In this step we provision another _Custom Resource_ which calls the Config Manager _Lambda_ function. But this time, the _Lambda_ function triggers a build in _CodeBuild_ where we, on the fly, execute a _kubectl_ command to update the _aws-auth ConfigMap_ in _Kubernetes_ with right privileges for standard users and groups in your organisation.

This completes provisioning a new EKS Cluster with all required configuration applied.

**STEP 0: Wait, what?**
Actually we need something very important before performing all these steps above. It is the right time to explain that now.

We need a Service Role with multiple service principals as shown in the following _CloudFormation_ template snippet:

```
Resources:
  EKSFactoryServiceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub EKSFactory-${AWS::Region}-ServiceRole
      Description: EKS Factory Service Role used by different services e.g. Service Catalog, CloudFormation, and CodeBuild for managing EKS environments.
      AssumeRolePolicyDocument:
        Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service:
            - cloudformation.amazonaws.com
            - codebuild.amazonaws.com
            - servicecatalog.amazonaws.com
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/AdministratorAccess
```

We will create this role as part of the solution deployment which is discussed later. We need this role before provisioning anything because _CloudFormation_, _CodeBuild_ and Service Catalog (which you will see later) services will use the same role.

### STACK 2. EKS Data Plane/Node Group

Once an EKS environment is created and the Cluster is provisioned, then we can provision one/more Data Plane (Node Group) using this stack. Following is the workflow of different resources being provisioned in this stack. As before, a child stack is created in each of these steps.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7u5b3qy2jx1qtn12lzpz.png)

**STEP 1: Provision Prerequisites**
First, we create a _Node IAM Role_ required for the Node Group.

**STEP 2: Register Node IAM Role**
Then, we deploy a _Custom Resource_ which make a call to the Config Manager (_Lambda_ function) and triggers a build in _CodeBuild_ to execute a _kubectl_ command to add the _Node IAM Role_ in the _aws-auth ConfigMap_ in _Kubernetes_. This is required for the nodes to be able to join the cluster.

**STEP 3: Provision Node Group**
The EKS Node Group is provisioned in this step. But before that, it creates a Launch Template based on which the Node Group is provisioned.

**STEP 4: Apply Labels and Tags**
We deploy a _Custom Resource_ at this stage that makes a call to the Config Manager (_Lambda_ function) to add any additional _Kubernetes_ labels to the Node Group and the tags to the underlying Auto-Scaling Group (ASG). The tags added at this stage to the ASG do not propagate to the already launched EC2 instances under the ASG, hence the _Lambda_ function also adds the tags to directly to those existing instances as well.

Finally, we get a new Node Group provisioned with the underlying nodes joined the EKS Cluster.

## Making it self-service

At this point, we are good in terms of provisioning Control Plane and Data Plane and all the underlying resources with all the automation in place as long as we have the _CloudFormation_ template files available with us. Now, if we package them as Service Catalog Products then end users can consume them in a self-service manner.

## Solution Deployment

The solution is available in the [Github](https://github.com/chandranathmondal/eks-factory) and can be deployed in one-click by executing [install.sh](https://github.com/chandranathmondal/eks-factory/blob/main/install.sh) as described in the [README.md](https://github.com/chandranathmondal/eks-factory/blob/main/README.md) file. It creates a Service Catalog Portfolio called _EKS Factory_ and deploys the _EKS Control Plane_ and _EKS Data Plane_ as two separate products. The solution also creates the EKSFactoryServiceRole (as discussed before) with multiple service principals so that other services (_CloudFormation_, _CodeBuild_ and Service Catalog) can use the same role.

> Note: If we deploy the solution in Service Catalog, technically we don’t need the CloudFromation service principal in the EKSFactoryServiceRole. But I kept it in my source code just in case you don’t use Service Catalog and provision all the resources directly via _CloudFormation_.

### Key Consideration before deployment

Update the [aws-auth.yml.template](https://github.com/chandranathmondal/eks-factory/blob/main/build-scripts/aws-auth.yml.template) based on your requirement. You need to add IAM users/roles that will have access to the provisioned clusters. Typically, these are the standard users/roles such as system administrators, audit users who would need to have access to all the resources. You can define different permission for different users/roles as shown below:

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::AWSAccountID:role/EKS-Admin
      username: EKS-Admin
      groups:
      - system:masters
    - rolearn: arn:aws:iam::AWSAccountID:role/EKS-Users
      username: EKS-Users
  mapUsers: |
    - userarn: arn:aws:iam::AWSAccountID:user/Audit-User
      username: audit-user
      groups:
      - reporting:read
```

## Key Features of the Solution

### 1. Self-service

The entire infrastructure can be provisioned, upgraded and decommissioned in a self-service manner with no manual intervention at the backend. This brings accuracy and agility.

### 2. IaC

Infrastructure is provisioned using declarative _CloudFormation_ templates, and “CloudFormation-first” approach. This makes it very easy for both developers as well as SysAdmins to maintain the environment going forward.

You can provision a whole infrastructure (an EKS Cluster with two Node Groups) using a simple yaml template like this:

```
AWSTemplateFormatVersion: 2010-09-09
Description: Template to create an EKS Cluster and two Node Groups.

Resources:
  Cluster:
    Type: AWS::ServiceCatalog::CloudFormationProvisionedProduct
    Properties:
      ProductName: EKS Control Plane
      ProvisionedProductName: my-prod-EKS-control-plane
      ProvisioningArtifactName: New Cluster
      ProvisioningParameters:
      - Key: EnvironmentName
        Value: my-prod
      - Key: KubernetesVersion
        Value: "1.21"
      - Key: KmsKeyArn
        Value: <YOUR KMS KEY ARN>
      - Key: VpcId
        Value: <YOUR VPC ID>
      - Key: Subnets
        Value: "<YOUR SUBNET IDs SEPARATED BY COMMAS>"
      - Key: VpcCniVersion
        Value: v1.10.1-eksbuild.1
      - Key: CoreDnsVersion
        Value: v1.8.4-eksbuild.1
      - Key: KubeProxyVersion
        Value: v1.21.2-eksbuild.2
        
NodeGroup1:
    Type: AWS::ServiceCatalog::CloudFormationProvisionedProduct
    Properties:
      ProductName: EKS Data Plane
      ProvisionedProductName: my-prod-EKS-121-141-NodeGroup-1
      ProvisioningArtifactName: New Node Group
      ProvisioningParameters:
      - Key: EnvironmentName
        Value: my-prod
      - Key: KubernetesVersionNumber
        Value: 121
      - Key: BuildNumber
        Value: 141
      - Key: NodeGroupName
        Value: NodeGroup-1
      - Key: AmiId
        Value: <YOUR AMI ID>
      - Key: CapacityType
        Value: ON_DEMAND
      - Key: InstanceType
        Value: m5.2xlarge
      - Key: VolumeSize
        Value: 60
      - Key: VolumeType
        Value: gp3
      - Key: MinSize
        Value: 1
      - Key: MaxSize
        Value: 3
      - Key: DesiredSize
        Value: 3
      - Key: Subnets
        Value: "<YOUR SUBNET IDs SEPARATED BY COMMAS>"
      - Key: EnableCpuCfsQuota
        Value: "true"
      - Key: HttpTokenState
        Value: optional
    DependsOn:
    - Cluster
  
  NodeGroup2:
    Type: AWS::ServiceCatalog::CloudFormationProvisionedProduct
    Properties:
      ProductName: EKS Data Plane
      ProvisionedProductName: my-prod-EKS-121-141-NodeGroup-2
      ProvisioningArtifactName: New Node Group
      ProvisioningParameters:
      - Key: EnvironmentName
        Value: my-prod
      - Key: KubernetesVersionNumber
        Value: 121
      - Key: BuildNumber
        Value: 141
      - Key: NodeGroupName
        Value: NodeGroup-2
      - Key: AmiId
        Value: <YOUR AMI ID>
      - Key: CapacityType
        Value: ON_DEMAND
      - Key: InstanceType
        Value: m5.2xlarge
      - Key: KeyPair
        Value: <YOUR KEY PAIR NAME>
      - Key: VolumeSize
        Value: 60
      - Key: VolumeType
        Value: gp3
      - Key: MinSize
        Value: 1
      - Key: MaxSize
        Value: 3
      - Key: DesiredSize
        Value: 3
      - Key: Subnets
        Value: "<YOUR SUBNET IDs SEPARATED BY COMMAS>"
      - Key: EnableCpuCfsQuota
        Value: "true"
      - Key: HttpTokenState
        Value: optional
      - Key: TaintKey
        Value: <YOUR TAINT KEY>
      - Key: TaintValue
        Value: <YOUR TAINT VALUE>
      - Key: TaintEffect
        Value: <YOUR TAINT EFFECT>
      - Key: AdditionalLabels
        Value: "label1=value1,label2=value2"
      - Key: AdditionalTags
        Value: "tag1=value1,tag2=value2"
    DependsOn:
    - Cluster
```

### 3. Integrated Build Platform

_CodeBuild_ enables us to perform any kind of (pre, post deployment) configuration and/or deploy workloads into the environment.

### 4. Integrated Business Logic Layer

_Lambda_ function enables us to write any custom component/logic and integrate features in a plug-and-play manner.

### 5. Serverless

The solution is completely serverless which is highly available, scalable and cost effective approach compared to any traditional model where we might need dedicated servers to run the different components such as Jenkins.

### 6. Cloud-native

The solution doesn’t use any third tools. This is good for organisations using AWS and prefer cloud-native solutions.

### 7. Flexible

The solution is flexible, components can be updated separately to add new features. For example, we can use it to deploy Cluster Autoscaler or applications such as monitoring tools post provisioning the cluster. All we have to do is prepare the steps in a shell script file and invoke a build with the script.

## Default behaviour of the Solution

### a. Separate Node IAM Role for each Node Group

We are creating a separate _Node IAM Role_ for each Node Group. This is based on the assumption that there could be requirement for different Node Groups to have different permissions. In that case, you can modify the _Node IAM Roles_ post provisioning the Data Plane (Node Group).

### b. Default labels and tags

We are adding _environmentName_ both as a _Kubernetes_ label and EC2 tag to the underlying nodes. But you can also provide any no. of additional labels and tags per Node Group while provisioning them.

### c. Zero/One Taint per Node Group

The Node Group template [eks-node-group.yaml](https://github.com/chandranathmondal/eks-factory/blob/main/eks-data-plane/eks-node-group.yaml) allows zero or one Taint to be applied to the underlying nodes. If you want more Taints to be applied then we can easily extend the solution to implement that similar to how it’s done for labels and tags.

## Taking the Solution to next level

Many organisations use ServiceNow for their ITIL process. We can enable end users to request EKS resources (Cluster, Node Group) via _ServiceNow_.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y8tehja25wfhmka01so3.jpeg)
The diagram is from official website. We are only using AWS Service Catalog in our solution. AWS Config and AWS Systems Manager are not in use.

This is achieved by using _AWS Service Management ConnectorforServiceNow_ (formerly the _AWS Service Catalog Connector_). The connector allows ServiceNow end users to provision, manage, and operate AWS resources natively through ServiceNow.

--------------------------------------------------------------

_How did you find this solution? Feel free to share your thoughts in the comments below._