Lessons in AWS Python CDK: 2-Parameterizing Your Stack
Tags: aws, python, beginners, tutorial
Published: 2022-03-20 13:40:23

{% details TLDR %}
I don't like using the `cdk.context.json` file. I use a `config/<env>.yaml` file that holds environment specific values and omegaconf to parse it.

Say you have dev and prod VPCs that already exist and you want to use the same stack to deploy to both. 

Files:

config/dev.yaml

```yaml
vpc:
  vpc_id: vpc-1234567890
```

config/prod.yaml

```yaml
vpc:
  vpc_id: vpc-9876543210
```

In the stack:

```python
from omegaconf import Omegaconf

deploy_env = "dev" # <-- This could/should be set by your CICD pipeline
conf = OmegaConf.load("config/{0}.yaml".format(deploy_env))

# load pre-existing vpc into variable
vpc = ec2.Vpc.from_lookup(self, conf.vpc.vpc_id, vpc_id=conf.vpc.vpc_id)
```

In typical yaml fashion everything is a dictionary or a list and you can, therefore, navigate and loop. (details below)
{% enddetails %}

{% details About Me %}
My name is Jakob and I am a DevOps Engineer. I used to be a lot of other things as well (Dish Washer, Retail Employee, Camp Counselor, Army Medic, Infectious Disease Researcher), but now I am a DevOps Engineer. I received no formal CS education but I'm not self taught, because I had thousands of instructors who taught me through their tutorials and blog posts. The culture of information sharing within the software engineering community is vital to everyone, especially those like me who didn't have other options. So, as I learn new things I will be documenting them through the eyes of someone learning for the first time, because those are the people most in need of a guide. Happy Learning! And don't be a stranger.
{% enddetails %}

---

Note: I am NOT going to be sourcing and fact checking everything here. This is not an O'Reilly book. The language and descriptions are intended to allow beginners to understand. If you want to be pedantic about details or critique my framing of what something is or how it works feel free to write your own post about it. These posts are intended to be enough to get you started so that you can begin breaking things in new ways on your own!

---
## The Problem ##

One thing that I found some difficulty with when I started using the AWS-CDK was how to handle deploying into multiple pre-existing environments. Of course, the CDK makes it easy to create a stand-alone stack with a new VPC, subnets, buckets, and certificates, but sometimes we have a pre-existing environment we need to deploy into. Or perhaps we want to use variations on the same stack to deploy into multiple environments, eg. smaller instances instances for a development environment. 

In Terraform, we might pass in a `dev.tfvars` or `prod.tfvars` file. With the CDK you can use the recommended `cdk.context.json` file and pass context dependent parameters into the stack. But after scanning over the documentation for how to add values to the context file, I decided it was too annoying and I wanted a better way.  

## My Solution ##

I have settled on pairing a yaml parsing library called `omegaconf` to make my own `.tfvars`-like parameter file. Let me show you how it works. 

### The Setup ###

First, create yourself come configuration files. I made a folder at the root of the project called `/config` where I created yaml files named for each environment, eg. `dev.yaml`, `uat.yaml`, `prod.yaml`. Lets also add a couple things to `dev.yaml`

```yaml
aws:
  account: "12345678910"
  region: us-east-1
env: Dev

```

Now, throw `omegaconf` into your requirements.txt file and `pip install` it. (you are using a virtual environment right?)

Import it into both the `app.py` file at the root of your project and any stack files you plan on using the parameters.

```python
from omegaconf import Omegaconf
```
---
### Implementation ###

Depending on how you are going to manage deploying to your environments, you are going to load a different config file. 

For example, if you are just going to deploy from your local computer, you can set an variable the the deploy environment at the top of your file (so that it is visible and easy to change) and then use that variable to load your config.

```python
 1   import aws_cdk as cdk
 2   from uber_for_cats.uber_for_cats_stack import UberForCatsStack
 3   from omegaconf import Omegaconf
 4   
 5   deploy_env = "dev"
 6   
 7   conf = OmegaConf.load("config/{0}.yaml".format(deploy_env))
 8   
 9   app = cdk.App()
10   UberForCatsStack(app, "UberForCats{0}".format(conf.env),
11       env=cdk.Environment(account=conf.aws.account, region=conf.aws.region),
12       )
13   
14   app.synth()
```

On `lines 5-7` the environment is set to dev and the `config/dev.yaml` file is set to `conf`.

If you were using a CICD pipeline to automatically deploy, `deploy_env` could be set based on a pipeline variable and `line 5` could look like this.

```python 
 5   deploy_env = os.getenv("DEPLOY_ENV")
```

But the result will be the same the values set in the `config/dev.yaml` file will be used and the file will be read as if the strings were there. 

```python
 1   import aws_cdk as cdk
 2   from uber_for_cats.uber_for_cats_stack import UberForCatsStack
 3   from omegaconf import Omegaconf
 4   
 5   deploy_env = "dev"
 6   
 7   conf = OmegaConf.load("config/dev.yaml")
 8   
 9   app = cdk.App()
10   UberForCatsStack(app, "UberForCatsDev",
11       env=cdk.Environment(account="12345678910", region="us-east-1"),
12       )
13   
14   app.synth()
```

This allows you to set other values in `config/uat.yaml` such as a different account, different sized instances/EBS volumes, autoscaling rules, etc. depending on what is required in each environment. 

### Something to Remember ###

Yaml is basically a nested dictionary that can also contain lists. When you see a `-` before something it is a list and therefore iterable. Take, for example, the following.

```yaml
aws:
  account: "12345678910"
  region: us-east-1
vpc: 
  vpc_id: vpc-aabbccdd
  subnet:
    private:
    - subnet-65asdf651sadf65
    - subnet-c65as1df65f56sa
    - subnet-afas65df1a6sdf5
```

Those private subnet IDs are a list and you can iterate over them. 

```python
private_subnets = []
for i, subnet in enumerate(conf.vpc.subnet.private):
    private_subnets.append(
        ec2.Subnet.from_subnet_id(self, "pri{0}".format(i), subnet_id=subnet)
    )
```
This would create a list of subnet objects (`ISubnet`) that you can use for the placement of an autoscaling group or EKS cluster. Neat!

Congrats, Those are the basics!  You should be able to get started. 

---
## Some Additional Tricks ##

While using a yaml file for configuration settings, I ran into some situations that might be worth sharing. 

### __Sometimes strings require extra steps__ ###

I am using the CDK to make some EKS clusters. Part of this process is creating node-groups (think autoscaling groups for Kubernetes). I wanted to leverage some spot instances for a portion of our development cluster and part of the process of creating these nodegroups in CDK is specifying the compute size and class. Because this could be different between environments I put it in the config file. 

```yaml
node_group:
  spot:
    min: 1
    max: 5
    type:
    - i_class: BURSTABLE3
      i_size: LARGE
    - i_class: BURSTABLE3
      i_size: XLARGE
    - i_class: COMPUTE6_INTEL
      i_size: LARGE
    - i_class: COMPUTE6_INTEL
      i_size: XLARGE
```

But that won't work for setting instance types:

```python
instance_type = ec2.InstanceType.of(
    ec2.InstanceClass.conf.node_group.spot.type[0].i_class,
    ec2.InstanceSize.conf.node_group.spot.type[0].i_size,
    )
```
And, honestly, it isn't very readable either. My workaround was making dictionaries of instance classes and sizes. then using the value in the yaml as the key the the appropriate class/size.

```yaml
node_group:
  spot:
    min: 1
    max: 5
    type:
    - i_class: t3
      i_size: large
    - i_class: t3
      i_size: xl
    - i_class: c6i
      i_size: large
    - i_class: c6i
      i_size: xl
```

```python
ec2_class = {
    "t3": ec2.InstanceClass.BURSTABLE3,  # max 2xl
    "c6i": ec2.InstanceClass.COMPUTE6_INTEL,  # min large
}

ec2_size = {
    "large": ec2.InstanceSize.LARGE,
    "xl": ec2.InstanceSize.XLARGE,
}
```

With the combination of the above, we can now use the string in the yaml as the key to pull in the ec2 objects in the format that the CDK requires. Below I make a list of the specified instance combinations and pass it into the EKS cluster as nodegroup capacity, but you could just as easily specify a list of classes and types and use a python library like `itertools` to make ALL of the combinations in one line. That might sacrifice readability though. So actually, don't do that. But you could...

```python
spot_instance_types = []
for instance_type in node_group.spot.type:
    this_type = ec2.InstanceType.of(
        ec2_class[instance_type.i_class], 
        ec2_size[instance_type.i_size],
    )
    spot_instance_types.append(this_type)

cluster.add_nodegroup_capacity(
    "{0}-spot-nodegroup".format(conf.env),
    nodegroup_name="{0}-spot-ng".format(conf.env),
    capacity_type=eks.CapacityType.SPOT,
    min_size=node_group.spot.min,
    max_size=node_group.spot.max,
    instance_types=spot_instance_types, # <-- list of instance types
    disk_size=250,
    subnets=ec2.SubnetSelection(subnets=private_subnets), # <-- those subnets from before!
)
```

---
### __Booleans are Useful__ ###

So we are promoting this project out to production and _someone_ doesn't think spot instances are a good idea even though you have diversified your spot pools. Throw a trigger into the config. 

```yaml
node_group:
  spot:
    enabled: False
    min: 1
    max: 5
```

Then you can run your node creation based on it!

```python
if conf.node_group.spot.enabled:
    spot_instance_types = []
    for instance_type in node_group.spot.type:
        this_type = ec2.InstanceType.of(
            ec2_class[instance_type.i_class], 
            ec2_size[instance_type.i_size],
        )
        spot_instance_types.append(this_type)

    cluster.add_nodegroup_capacity(
        "{0}-spot-nodegroup".format(conf.env),
        nodegroup_name="{0}-spot-ng".format(conf.env),
        capacity_type=eks.CapacityType.SPOT,
        min_size=node_group.spot.min,
        max_size=node_group.spot.max,
        instance_types=spot_instance_types, # <-- list of instance types
        disk_size=250,
        subnets=ec2.SubnetSelection(subnets=private_subnets), # <-- those subnets from before!
    )
```
Beautiful!