Connecting AWS ALB to Nginx Ingress with Terraform #EKS
Published: 2023-06-07 17:28:54

By default, for [reasons](https://github.com/kubernetes/ingress-nginx/issues/6292#issuecomment-728974161) I don't care about anymore, you can't really use ALB in OOTB fashion with [Kubernetes Nginx Ingress](https://github.com/kubernetes/ingress-nginx).

In this post I will show you how to create ALB using Terraform and set it up with EKS cluster and Kubernetes Nginx controller.

Prerequisites:
- EKS Cluster with NodeGroup
- AWS Account access (yeah)
- K8S knowledge, I'm not gonna debug your yaml identation errors


## How does it work?

We create Application LoadBalancer, we use LB Target Groups with `target_type = "instance"`, then we set up our listeners to forward to traffic to given TargetGroup aaaand last step is attaching the autoscaling group (EKS worker nodes) to target groups. Ingress controller then listens on high ports and properly receives and distributes all the traffic.

Let's break that down!

## Loadbalancer parkour

Let's start with simple Security Group. As it's a good practice, we'll use `data` lookups in Terraform to find suitable subnets and VPC.

For this article I have created `dev-eks` VPC and `dev-eks` cluster in it, with 3 private and 3 public subnets. You can use official [VPC module](https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest)

```terraform
data "aws_vpc" "dev_vpc" {
  tags = {
    Name = "dev-eks"
  }
}

data "aws_subnets" "dev_private_subnets" {
  filter {
    name   = "tag:kubernetes.io/cluster/dev-eks"
    values = ["shared"]
  }

  filter {
    name   = "tag:kubernetes.io/role/internal-elb"
    values = ["1"]
  }
}

data "aws_subnets" "dev_public_subnets" {
  filter {
    name   = "tag:kubernetes.io/cluster/dev-eks"
    values = ["shared"]
  }

  filter {
    name   = "tag:kubernetes.io/role/elb"
    values = ["1"]
  }
}
```
 
This is really comfy way of getting the info about objects you interact with in your code without too much unnecessary hardcoding.

Now, the Security Group:

```terraform
resource "aws_security_group" "allow_https" {
  name        = "dev_allow_tls"
  description = "Allow HTTP/TLS inbound traffic"
  vpc_id      = data.aws_vpc.dev_vpc.id

  ingress {
    description = "HTTPS from internet"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTP from internet"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  
  ingress {
    description = "dev"
    from_port   = 0
    to_port     = 0
    protocol    = "all"
    cidr_blocks = [data.aws_vpc.dev_vpc.cidr_block]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  tags = {
    Name = "alb-sg-dev"
  }
}
```

The traffic will flow like this:

**HTTP**:
:alien: -> Loadbalancer _(port :80)_ -> EKS Worker nodes in NodeGroup _(port :30080)_ -> ingress controller (we'll get to that part soon)

**HTTPS**:
:alien: -> Loadbalancer _(port :443)_ -> EKS Worker nodes in NodeGroup _(port :30443)_ -> ingress controller

I'm too lazy to do a proper diagram. Imagine one is here. 


### Loadbalancer 
Now we can create LoadBalancer! The setup is really nothing extraordinary.

```terraform
resource "aws_lb" "alb_for_dev" {
  name               = "alb-ingress-dev-eks"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.allow_https.id]
  subnets            = data.aws_subnets.dev_public_subnets.ids

  enable_deletion_protection = true

  tags = {
    Environment = "dev"
    Terraform   = "true"
  
  }
}
```

While you are here consider adding `access_logs` configuration with S3. You might want to use it at some point.

### Target Groups

TGs route requests to all registered targets within TargetGroup. It can be instance, IP (?) or lambda. Read more [here](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html). 

We are gonna use `instance` target type here. Of course we need two TGs, for HTTP and HTTPs. 

```terraform
resource "aws_lb_target_group" "http" {
  name        = "eks-dev-http"
  target_type = "instance"
  port        = "30080"
  protocol    = "HTTP"
  vpc_id      = data.aws_vpc.dev_vpc.id

  health_check {
    path     = "/"
    port     = "30080"
    protocol = "HTTP"
    matcher  = "200,404"
  }

  tags = {
    Environment = "dev"
  }
}

resource "aws_lb_target_group" "https" {
  name        = "eks-dev-https"
  target_type = "instance"
  port        = "30443"
  protocol    = "HTTPS"
  vpc_id      = data.aws_vpc.dev_vpc.id

  health_check {
    path     = "/"
    port     = "30443"
    protocol = "HTTPS"
    matcher  = "200,404"
  }

  tags = {
    Environment = "dev"
  }
}
```

This example does not have any lucrative healthchecks on this level. Adjust the code to your needs, or don't, I don't care :shrug: 

### Listeners

This is also important - we need to tell our ALB that it must listen on standard HTTP and HTTPS port. You need ACM cert for that also. 

```terraform
resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.alb_dev.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.http.arn
  }
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.alb_dev.arn
  port              = "443"
  protocol          = "HTTPS"
  certificate_arn   = "arn:aws:acm:us-east-4:4206969777:certificate/whoa-some-id-was-here"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.https.arn
  }
}
```

And now we attach ASG to LoadBalancer's Target Group. I have module for adding NodeGroups so I have module output there, but for simplicity we'll just imagine that 'some-asg-name-123' is valid and proper ASG. 

```terraform
resource "aws_autoscaling_attachment" "asg_alb_attachment_dev_http" {
  autoscaling_group_name = "some-asg-name-123"
  lb_target_group_arn    = aws_lb_target_group.http.arn
}

resource "aws_autoscaling_attachment" "asg_alb_attachment_dev_https" {
  autoscaling_group_name = "some-asg-name-123"
  lb_target_group_arn    = aws_lb_target_group.https.arn
}
```

Optionally at this level I recommend adding Route53 record, just A type record that adds custom entry to your hosted zone. If you need to replace LBs or do any shady operation it's easier to update record and easily revert changes if something catches on fire. 

```terraform
resource "aws_route53_record" "lb" {
  zone_id = "Z035554117EW97P3ONW5G"
  name    = "dev-lb.somedomain.com"
  type    = "A"

  alias {
    name                   = aws_lb.alb_dev.dns_name
    zone_id                = aws_lb.alb_dev.zone_id
    evaluate_target_health = false
  }
}
output "alb_dev_url" {
  value = aws_lb.alb_dev.dns_name
}
```

And hopefully if you've been careful enough during this relentless copy paste, Terraform code should be valid. Remember to use `terraform fmt`! 

Now we need stuff to listen on 30443 and 30080 on EKS, and that must be Ingress controller. 

### Controller setup 

This is fairly easy, install Nginx like you would normally but apply following diff in `values.yaml` or however your vars are called. 

```yaml
controller:
  service:
    annotations:
      external-dns.alpha.kubernetes.io/hostname: <url-you-added-to-route53-in-previous-step>.com
    type: NodePort
    nodePorts:
      http: 30080 
      https: 30443
    externalTrafficPolicy: Cluster

  ingressClassResource:
    default: true
```

How does this work is all Nodes in given NodeGroup reserve Ports 30080 and 30443 and listen on those, and forward requests to Ingress Controller. Then Nginx based on the Host header routes request to proper pods. 

# My comments

I _highly_ recommend getting that R53 entry and making it work with External DNS if you don't do it already. 

One of limitations in the current code is that you attach single ASG from NodeGroup to LB's Target Group. If your cluster has more NodeGroups, you either schedule Nginx on selected Nodes or do some `for_each` shenanigans with Terraform. I have single NodeGroup so I have no issues. 

The same thing should work with any other Ingress controllers, you just need to make it use NodePort. 

Cheers!