AWS EKS FluxCD Lab Notes
Published: 2023-12-03 15:35:23

## Sources

* [Github Repo](https://github.com/junglekid/aws-eks-fluxcd-lab)

* [Article 1 of 3](https://www.linkedin.com/pulse/using-flux-gitops-tool-amazon-elastic-kubernetes-service-rasmuson/) - Dallin Rasmuson

* [Article 2 of 3](https://www.linkedin.com/pulse/using-flux-gitops-tool-amazon-elastic-kubernetes-service-rasmuson-1c/) - Dallin Rasmuson

* [Article 3 of 3](https://www.linkedin.com/pulse/using-flux-gitops-tool-amazon-elastic-kubernetes-service-rasmuson-1f/) - Dallin Rasmuson

## Part 1

### Install and Configure AWS CLI

* [Install CLI V2](https://awscli.amazonaws.com/AWSCLIV2.pkg)

* `aws --v` verify version 2

* Configure AWS using SSO as described [here](https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html)

* Open [AWS Identity Center](console.aws.amazon.com/singlesignon/home)

* Create User and add Administrative Permissions.

* Take note of the `start` url.

```bash
aws configure sso
SSO session name (Recommended): terraform
SSO start URL [None]: https://d-REDACTED.awsapps.com/start
SSO region [None]: us-east-1
SSO registration scopes [sso:account:access]: sso:account:access
```

### Install Terraform

* `brew install terraform`

* `terraform -install-autocomplete`

* `alias tf='terraform'`

### Install Helm

* `brew install helm`

### Install kubectl

* `brew install kubernetes-cli`

* `alias k='kubectl'`

### Github Personal Access Token (PAT)

If you want to bootstrap Flux for a repository owned by a GitHub organization, it is recommended to create a dedicated user for Flux under your organization as described [here](https://fluxcd.io/flux/installation/bootstrap/github/#github-organization)

* I used the `+` operator to specify a unique email tht is still delivered to my normal email: myemail+flux@mydomain.com

* https://github.com/settings/tokens

### AWS Resources

We need to create a few AWS resources that will comprise the Terraform backend.  Terraform stores state and locks using S3 and DynamoDB.  Create these through the web console:

S3 Bucket

* Bucket: {project_name}-terraform-backend

DynamoDB Table

* Table: {project_name}-terraform-lock-table

* Partition Key: LockID

Route53

* Register or transfer a domain

### Update Locals

* Update locals.tf and provider.tf using info from above.  

* Note that the `public_domain` should match `Hosted zone name` which is likely your domain name free of a any subdomains.

### Init and Apply

```bash
cd terraform
terraform init
terraform validate
terraform plan -out=plan.out
terraform apply plan.out
```  

### Troubleshooting

First errors were

```
│ Error: Unauthorized
│
│   with kubernetes_service_account.alb_service_account,
│   on eks_roles.tf line 18, in resource "kubernetes_service_account" "alb_service_account":
│   18: resource "kubernetes_service_account" "alb_service_account" {
│
╵
╷
│ Error: Unauthorized
│
│   with kubernetes_service_account.external_dns_service_account,
│   on eks_roles.tf line 57, in resource "kubernetes_service_account" "external_dns_service_account":
│   57: resource "kubernetes_service_account" "external_dns_service_account" {
│
╵
╷
│ Error: Unauthorized
│
│   with kubernetes_service_account.cluster_autoscaler_service_account,
│   on eks_roles.tf line 96, in resource "kubernetes_service_account" "cluster_autoscaler_service_account":
│   96: resource "kubernetes_service_account" "cluster_autoscaler_service_account" {
```

Steps to Resolve

* [Specialized Terraform Cloud Engineer GPT](https://chat.openai.com/g/g-Z1MNFH9LH-terraform-cloud-engineer) - This Terraform specialized AI suggested the kubectl wasn't configured.  And that the configuration is specified in the kubernetes provider.

```tf
provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    # This requires the awscli to be installed locally where Terraform is executed
    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}
```

Armed with this, I noticed the args didn't include `--profile jeff` and so it's possible this setup isn't in tune with my way of authenticating with SSO.  For example:

* `aws sts get-caller-identity` doesn't work

* `aws sts get-caller-identity --profile jeff` does work

So I asked [Specialized Terraform Cloud Engineer GPT](https://chat.openai.com/g/g-Z1MNFH9LH-terraform-cloud-engineer) how to fix it and they recommended editing `~/.aws/config` such that the defaults of the `[jeff]` profile were moved to `[default]`

```
[default]
sso_session = terraform
sso_account_id = REDACTED
sso_role_name = AdministratorAccess
region = us-east-1
```

Running the test now

* `aws sts get-caller-identity` works

Retrying

```bash
terraform plan -out=plan.out
terraform apply plan.out
```  

It worked!

## Part 2

* The guide is a cakewalk until the Bootstrap command.  To get it working in an organization instead of a personal github account, I had to remove the last `--personal`

* Also, be sure to rename the folder in clusters to {your-cluster-name}

```bash
flux bootstrap github \
  --components-extra=image-reflector-controller,image-automation-controller \
  --owner=$GITHUB_OWNER \
  --repository=$GITHUB_REPO_NAME \
  --private=false \
  --path=clusters/{your-cluster-name}
```

## More Troubleshooting

I wanted to try everything again and after I destroyed everything, I got this when I tried to bring it back up.

### Expired Token

`Error when retrieving token from sso: Token has expired and refresh failed`

Solution

`aws sso login`

### Pending Pods

* Increase nodes in cluster

### Unexpected State Data

I got a 

```
The checksum calculated for the state stored in S3 does not match the checksum
stored in DynamoDB.
```

So I scanned the db to see how it looked

```bash
aws dynamodb scan --table-name my-project-terraform-lock-table
```

Then I updated the digest in DynamoDB to the state it needed.

```bash
aws dynamodb put-item \
    --table-name my-project-terraform-lock-table \
    --item '{
            "LockID": {
                "S": "my-project-terraform-backend/dev-md5"
            },
            "Digest": {
                "S": "f53b84cddc4cfe4c7f868267cbdd2ab1"
            }
    }'
```

Back to normal.