Configuring an isolated network in AWS
Tags: aws, security, terraform, network
Published: 2021-03-20 15:00:21

In the [first part][part-1] we introduced the security patterns that could be implemented to secure the connectivity between Amazon EKS and Amazon RDS. In this part we will implement the network isolation by deploying the following AWS resources:

* VPC with eight subnets
  * 2 public and private subnets for Amazon EKS.
  * 2 public and private subnets for Amazon RDS.
* An Internet Gateway attached to the VPC.
* NAT gateways attached to the EKS public subnets.
* Network ACL for each couple of subnets.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/haaklaqq5hkfxox1wtyj.jpg)

# VPC

Let's start with the `Virtual Private Cloud`. 

Create a terraform file `infra/plan/vpc.tf`. A simple VPC resource is created:

```terraform
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr_block
  instance_tenancy     = "default"
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name        = "main-${var.env}"
    Environment = var.env
  }
}
```

# Subnets

Now we create our 8 subnets.

* Two public subnets for high-availability. They will host our External Application Load Balancers created by Amazon EKS and all internet facing Kubernetes workloads.
* Two private subnets for high-availability. They will host our Internal Application Load Balancers created by Amazon EKS and all internal Kubernetes workloads.
* (Optional) Two other public subnets for high-availability. They will host our External Network Load Balancers to expose our private RDS PostgreSQL instance.
* Two other private subnets for high-availability. They will host our Amazon RDS PostgreSQL instance.

Create a terraform file `infra/plan/subnet.tf`

```terraform
resource "aws_subnet" "private" {
  for_each = {
    for subnet in local.private_nested_config : "${subnet.name}" => subnet
  }

  vpc_id                  = aws_vpc.main.id
  cidr_block              = each.value.cidr_block
  availability_zone       = each.value.az
  map_public_ip_on_launch = false

  tags = {
    Environment = var.env
    Name        = "${each.value.name}-${var.env}"
    "kubernetes.io/role/internal-elb" = each.value.eks ? "1" : ""
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

resource "aws_subnet" "public" {
  for_each = {
    for subnet in local.public_nested_config : "${subnet.name}" => subnet
  }

  vpc_id                  = aws_vpc.main.id
  cidr_block              = each.value.cidr_block
  availability_zone       = each.value.az
  map_public_ip_on_launch = true

  tags = {
    Environment = var.env
    Name        = "${each.value.name}-${var.env}"
    "kubernetes.io/role/elb" = each.value.eks ? "1" : ""
  }

  lifecycle {
    ignore_changes = [tags]
  }
}
```

I used a local variable to differentiate between each type of subnet.

Create a terraform file `infra/plan/variable.tf`

```terraform

variable "private_network_config" {
  type = map(object({
      cidr_block               = string
      az                       = string
      associated_public_subnet = string
      eks                      = bool
  }))

  default = {
    "private-eks-1" = {
        cidr_block               = "10.0.0.0/23"
        az                       = "eu-west-1a"
        associated_public_subnet = "public-eks-1"
        eks                      = true
    },
    "private-eks-2" = {
        cidr_block               = "10.0.2.0/23"
        az                       = "eu-west-1b"
        associated_public_subnet = "public-eks-2"
        eks                      = true
    },
    "private-rds-1" = {
        cidr_block               = "10.0.4.0/24"
        az                       = "eu-west-1a"
        associated_public_subnet = ""
        eks                      = false
    },
    "private-rds-2" = {
        cidr_block               = "10.0.5.0/24"
        az                       = "eu-west-1b"
        associated_public_subnet = ""
        eks                      = false
    }
  }
}

locals {
    private_nested_config = flatten([
        for name, config in var.private_network_config : [
            {
                name                     = name
                cidr_block               = config.cidr_block
                az                       = config.az
                associated_public_subnet = config.associated_public_subnet
                eks                      = config.eks
            }
        ]
    ])
}

variable "public_network_config" {
  type = map(object({
      cidr_block              = string
      az                      = string
      nat_gw                  = bool
      eks                     = bool
  }))

  default = {
    "public-eks-1" = {
        cidr_block = "10.0.6.0/23"
        az = "eu-west-1a"
        nat_gw = true
        eks = true
    },
    "public-eks-2" = {
        cidr_block = "10.0.8.0/23"
        az = "eu-west-1b"
        nat_gw = true
        eks = true
    },
    "public-rds-1" = {
        cidr_block = "10.0.10.0/24"
        az = "eu-west-1a"
        nat_gw = false
        eks = false
    },
    "public-rds-2" = {
        cidr_block = "10.0.11.0/24"
        az = "eu-west-1b"
        nat_gw = false
        eks = false
    }
  }
}

locals {
    public_nested_config = flatten([
        for name, config in var.public_network_config : [
            {
                name                    = name
                cidr_block              = config.cidr_block
                az                      = config.az
                nat_gw                  = config.nat_gw
                eks                     = config.eks
            }
        ]
    ])
}
```

# Internet Gateway

To allow our public Subnets to communicate with the internet, we need to create an internet gateway and associate it to the public subnets using route tables.

Create a terraform file `infra/plan/igw.tf`

```terraform
resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Environment = var.env
    Name        = "igw-${var.env}"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Environment = var.env
    Name        = "rt-public-${var.env}"
  }
}

resource "aws_route_table_association" "public" {
  for_each = {
    for subnet in local.public_nested_config : "${subnet.name}" => subnet
  }

  subnet_id      = aws_subnet.public[each.value.name].id
  route_table_id = aws_route_table.public.id
}
```

# NAT gateways

In order to allow our private subnets used by Amazon EKS to access the internet, we need to create NAT Gateways on the public subnets used by Amazon EKS. We associate NAT Gateways with private subnets using route tables.

Create a terraform file `infra/plan/nat.tf`

```terraform
resource "aws_eip" "nat" {
  for_each = {
    for subnet in local.public_nested_config : "${subnet.name}" => subnet
    if subnet.nat_gw == true
  }
  
  vpc      = true

  tags = {
    Environment = var.env
    Name        = "eip-${each.value.name}-${var.env}"
  }
}

resource "aws_nat_gateway" "nat-gw" {
  for_each = {
    for subnet in local.public_nested_config : "${subnet.name}" => subnet
    if subnet.nat_gw == true
  }

  allocation_id = aws_eip.nat[each.value.name].id
  subnet_id     = aws_subnet.public[each.value.name].id

  tags = {
    Environment = var.env
    Name        = "nat-${each.value.name}-${var.env}"
  }
}

resource "aws_route_table" "private" {
  for_each = {
    for subnet in local.public_nested_config : "${subnet.name}" => subnet
    if subnet.nat_gw == true
  }

  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat-gw[each.value.name].id
  }

  tags = {
    Environment = var.env
    Name        = "rt-${each.value.name}-${var.env}"
  }
}

resource "aws_route_table_association" "private" {

  for_each = {
    for subnet in local.private_nested_config : "${subnet.name}" => subnet
    if subnet.associated_public_subnet != ""
  }

  subnet_id      = aws_subnet.private[each.value.name].id
  route_table_id = aws_route_table.private[each.value.associated_public_subnet].id
}
```

# Network Access Control List

Network ACL allows us to restrict the inbound and outbound network traffic to and from a subnet. In our case, we can implement the following rules:

* EKS private and public subnets, allow all inbound / outbound network traffic. We need to have these rules to allow Amazon EKS Control Plane to communicate with Worker nodes.
* RDS public subnets
 * allow all inbound / outbound TCP network traffic to RDS private subnets
 * allow tcp inbound / outbound TCP network traffic to a specific range of IP addresses only on the RDS port.
* RDS private subnet
 * allow inbound traffic on the RDS port from EKS private subnets and all TCP traffic from RDS private subnets.
 * allow all outgoing TCP network traffic to EKS private subnets and RDS public subnets.

Create a terraform file `infra/plan/nacl.tf`

```terraform
resource "aws_network_acl" "eks-external-zone" {
  vpc_id = aws_vpc.main.id

  subnet_ids = [aws_subnet.public["public-eks-1"].id, aws_subnet.public["public-eks-2"].id]

  tags = {
    Name        = "eks-external-zone-${var.env}"
    Environment = var.env
  }
}

resource "aws_network_acl_rule" "eks-ingress-external-zone-rules" {
  network_acl_id = aws_network_acl.eks-external-zone.id
  rule_number    = 100
  egress         = false
  protocol       = "-1"
  rule_action    = "allow"
  cidr_block     = "0.0.0.0/0"
  from_port      = 0
  to_port        = 0
}

resource "aws_network_acl_rule" "eks-egress-external-zone-rules" {
  network_acl_id = aws_network_acl.eks-external-zone.id
  rule_number    = 100
  egress         = true
  protocol       = "-1"
  rule_action    = "allow"
  cidr_block     = "0.0.0.0/0"
  from_port      = 0
  to_port        = 0
}

resource "aws_network_acl" "eks-internal-zone" {
  vpc_id = aws_vpc.main.id

  subnet_ids = [aws_subnet.private["private-eks-1"].id, aws_subnet.private["private-eks-2"].id]

  tags = {
    Name        = "eks-internal-zone-${var.env}"
    Environment = var.env
  }
}

resource "aws_network_acl_rule" "ingress-internal-zone-rules" {
  network_acl_id = aws_network_acl.eks-internal-zone.id
  rule_number    = 100
  egress         = false
  protocol       = "-1"
  rule_action    = "allow"
  cidr_block     = "0.0.0.0/0"
  from_port      = 0
  to_port        = 0
}

resource "aws_network_acl_rule" "egress-internal-zone-rules" {
  network_acl_id = aws_network_acl.eks-internal-zone.id
  rule_number    = 100
  egress         = true
  protocol       = "-1"
  rule_action    = "allow"
  cidr_block     = "0.0.0.0/0"
  from_port      = 0
  to_port        = 0
}

resource "aws_network_acl" "rds-external-zone" {
  vpc_id = aws_vpc.main.id

  subnet_ids = [aws_subnet.public["public-rds-1"].id, aws_subnet.public["public-rds-2"].id]

  tags = {
    Name        = "rds-external-zone-${var.env}"
    Environment = var.env
  }
}

locals {
  nacl_ingress_rds_external_zone_infos = flatten([{
      cidr_block = var.internal_ip_range
      priority   = 100
      from_port  = var.rds_port
      to_port    = var.rds_port
  }, {
      cidr_block = aws_subnet.private["private-rds-1"].cidr_block
      priority   = 101
      from_port  = 0
      to_port    = 65535
  },{
      cidr_block = aws_subnet.private["private-rds-2"].cidr_block
      priority   = 102
      from_port  = 0
      to_port    = 65535
  },{
      cidr_block = aws_subnet.public["public-rds-1"].cidr_block
      priority   = 103
      from_port  = 0
      to_port    = 65535
  },{
      cidr_block = aws_subnet.public["public-rds-2"].cidr_block
      priority   = 104
      from_port  = 0
      to_port    = 65535
  }]) 
}

resource "aws_network_acl_rule" "rds-ingress-external-zone-rules" {
  for_each  = {
    for subnet in local.nacl_ingress_rds_external_zone_infos : "${subnet.priority}" => subnet
  }

  network_acl_id = aws_network_acl.rds-external-zone.id
  rule_number    = each.value.priority
  egress         = false
  protocol       = "tcp"
  rule_action    = "allow"
  cidr_block     = each.value.cidr_block
  from_port      = each.value.from_port
  to_port        = each.value.to_port
}

resource "aws_network_acl_rule" "rds-egress-external-zone-rules" {
  network_acl_id = aws_network_acl.rds-external-zone.id
  rule_number    = 100
  egress         = true
  protocol       = "tcp"
  rule_action    = "allow"
  cidr_block     = "0.0.0.0/0"
  from_port      = 0
  to_port        = 65535
}

resource "aws_network_acl" "rds-secure-zone" {
  vpc_id = aws_vpc.main.id

  subnet_ids = [aws_subnet.private["private-rds-1"].id, aws_subnet.private["private-rds-2"].id]

  tags = {
    Name        = "rds-secure-zone-${var.env}"
    Environment = var.env
  }
}

locals {
  nacl_secure_ingress_egress_infos = flatten([{
      cidr_block = aws_subnet.private["private-eks-1"].cidr_block
      priority   = 101
      from_port  = var.rds_port
      to_port    = var.rds_port
  },{
      cidr_block = aws_subnet.private["private-eks-2"].cidr_block
      priority   = 102
      from_port  = var.rds_port
      to_port    = var.rds_port
  },{
      cidr_block = aws_subnet.private["private-rds-1"].cidr_block
      priority   = 103
      from_port  = 0
      to_port    = 65535
  },{
      cidr_block = aws_subnet.private["private-rds-2"].cidr_block
      priority   = 104
      from_port  = 0
      to_port    = 65535
  },{
      cidr_block = aws_subnet.public["public-rds-1"].cidr_block
      priority   = 105
      from_port  = 0
      to_port    = 65535
  },{
      cidr_block = aws_subnet.public["public-rds-2"].cidr_block
      priority   = 106
      from_port  = 0
      to_port    = 65535
  }]) 
}

resource "aws_network_acl_rule" "ingress-secure-zone-rules" {
  for_each  = {
    for subnet in local.nacl_secure_ingress_egress_infos : "${subnet.priority}" => subnet
  }

  network_acl_id = aws_network_acl.rds-secure-zone.id
  rule_number    = each.value.priority
  egress         = false
  protocol       = "tcp"
  rule_action    = "allow"
  cidr_block     = each.value.cidr_block
  from_port      = each.value.from_port
  to_port        = each.value.to_port
}

resource "aws_network_acl_rule" "egress-secure-zone-rules" {
  for_each  = {
    for subnet in local.nacl_secure_ingress_egress_infos : "${subnet.priority}" => subnet
  }
  network_acl_id = aws_network_acl.rds-secure-zone.id
  rule_number    = each.value.priority
  egress         = true
  protocol       = "tcp"
  rule_action    = "allow"
  cidr_block     = each.value.cidr_block
  from_port      = 0
  to_port        = 65535
}
```

Let's configure our terraform.

Complete the `infra/plan/variable.tf`

```terraform
variable "region" {
  type    = string
  default = "eu-west-1"
}

variable "az" {
  type    = list(string)
  default = ["eu-west-1a", "eu-west-1b"]
}

variable "env" {
  type = string
}

variable "vpc_cidr_block" {
  type = string
}

variable "internal_ip_range" {
    type = string
}
```

Add a `infra/plan/main.tf` file

```terraform
data "aws_caller_identity" "current" {}
```

Add a `infra/plan/version.tf` file

```terraform
terraform {
  required_version = ">= 0.12"
}
```

Add a `infra/plan/provider.tf` file

```terraform
provider "aws" {
  region = var.region
}
```

And a `infra/plan/backend.tf`

```terraform
terraform {
  backend "s3" {
  }
}
```

Now, export the following variables and create a bucket to save your terraform states.

```shell
export ENV=<ENV>
export REGION=eu-west-1
export EKS_CLUSTER_NAME=eks-cluster-$ENV
export AWS_PROFILE=<AWS_PROFILE>
export INTERNAL_IP_RANGE=<LOCAL_OR_INTERNAL_IP_RANGES>
export TERRAFORM_BUCKET_NAME=<BUCKET_NAME>

# Create bucket
aws s3api create-bucket \
     --bucket $TERRAFORM_BUCKET_NAME \
     --region $REGION \
     --create-bucket-configuration LocationConstraint=$REGION

# Make it not public     
aws s3api put-public-access-block \
    --bucket $TERRAFORM_BUCKET_NAME \
    --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Enable versioning
aws s3api put-bucket-versioning \
    --bucket $TERRAFORM_BUCKET_NAME \
    --versioning-configuration Status=Enabled
```

Create a `infra/envs/$ENV/terraform.tfvars` and deploy the infrastructure:

```shell
env               = "<ENV>"
vpc_cidr_block    = "10.0.0.0/16"
internal_ip_range = "<INTERNAL_IP_RANGE>"
az                = ["eu-west-1a", "eu-west-1b"]
```

```shell
cd infra/envs/dev

sed -i "s,<INTERNAL_IP_RANGE>,$INTERNAL_IP_RANGE,g; s,<ENV>,$ENV,g" terraform.tfvars

terraform init \
    -backend-config="bucket=$TERRAFORM_BUCKET_NAME" \
    -backend-config="key=$ENV/terraform-state" \
    -backend-config="region=$REGION" \
../../plan/ 

terraform apply ../../plan/ 
```

Let's check if all the resources have been created and are working correctly

VPC

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2hbxfhjah2w7ii6img8a.jpg)

Subnets

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d0qnmaz970x3bj7wc4yn.jpg)

Internet Gateway

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/94hduccvyockez5ggn2x.jpg)

NAT Gateways 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0su4zxvsiz7t133hf33w.jpg)

# Conclusion

Our network is now ready to host our AWS resources. In the [next part][part-3], we will focus on setting up Amazon EKS.

[part-1]: https://dev.to/stack-labs/securing-the-connectivity-between-amazon-eks-and-amazon-rds-part-1-527o
[part-3]: https://dev.to/stack-labs/securing-the-connectivity-between-amazon-eks-and-amazon-rds-part-3-405a