Grant Kubernetes Pods Access to AWS Services Using OpenID Connect
Tags: kubernetes, aws, ansible, opentofu
Published: 2024-04-22 09:41:53

---
title: Grant Kubernetes Pods Access to AWS Services Using OpenID Connect
published: true
date: 2024-04-22 08:42:46 UTC
tags: Kubernetes,AWS,Ansible,OpenTofu
canonical_url: https://developer-friendly.blog/2024/04/22/grant-kubernetes-pods-access-to-aws-services-using-openid-connect/
---

# Grant Kubernetes Pods Access to AWS Services Using OpenID Connect

Learn how to establish a trust relationship between a Kubernetes cluster and AWS IAM to grant cluster generated Service Account tokens access to AWS services using OIDC & without storing long-lived credentials.

*Originally published at [https://developer-friendly.blog](https://developer-friendly.blog/2024/04/22/grant-kubernetes-pods-access-to-aws-services-using-openid-connect/) on April 22, 2024.*

<!-- more -->
## Introduction

In our [previous post](https://developer-friendly.blog/2024/04/14/what-is-openid-connect-authentication-a-practical-guide/), we discussed what OpenID Connect (OIDC) is and how to use it to authenticate identities from one system to another.

We covered why it is crucial to avoid storing long-lived credentials and the benefits of employing OIDC for the task of authentication.

If you haven't read that one already, here's a recap:

- OIDC is an authentication protocol that allows the identities in one system to authenticate to another system.
- It is based on OAuth 2.0 and JSON Web Tokens (JWT).
- Storing long-lived credentials is risky and should be avoided at all cost if possible.
- OIDC provides a secure way to authenticate identities without storing long-lived credentials.
- It is widely used in modern applications and systems.
- The hard requirements is that both the Service Provider and the Identity Provider must be OIDC compliant.
- With OIDC you will only keep the identities and their credentials in one system and authenticate them to another system without storing any long-lived credentials. The former is called the Identity Provider and the latter is called the Service Provider.

We also covered a practical example of authenticating GitHub runners to AWS IAM by establishing a trust relationship between GitHub and AWS using OIDC.

In this post, we will take it one step further and provide a way for the pods of our Kubernetes cluster to authenticate to AWS services using OIDC.

This post will provide a walkthrough of granting such access to a bear-metal Kubernetes cluster ([k3s](https://docs.k3s.io/)) using only the power of OpenID Connect protocol. In a later post, we'll show you how easy it is to achieve the same with a managed Kubernetes cluster like [Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/). But, first let's understand the fundamentals by trying it on a bear-metal cluster.

We will not store any credentials in our pods and as such, won't ever have to worry about other security concerns such as secret rotations!

With that intro out of the way, let's dive in!

## Prerequisites

Make sure you have the following prerequisites in place before proceeding:

- A Kubernetes cluster that can be exposed to the internet.
- An AWS account to create an OIDC provider and IAM roles.
- A verified root domain name that YOU own. Skip this if you're using a managed Kubernetes cluster.
- [OpenTofu v1.6](https://github.com/opentofu/opentofu/releases/tag/v1.6.2)
- [Ansible v2.16](https://github.com/ansible/ansible/releases/tag/v2.16.6)

## Roadmap

Let's see what we are trying to achieve in this guide.

Our end goal is to [create an Identity Provider (IdP) in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html). After doing so, we will be able to create an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) with a trust relationship to the IdP.

**Ultimately, the pods in our Kubernetes cluster that have the desired [Service Account(s)](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) will be able to talk to the AWS services**.

To achieve this, and as per the OIDC specification, the following endpoints must be exposed through an HTTPS endpoint with a [verified TLS certificate](https://openid.net/specs/openid-connect-core-1_0.html):

- `/.well-known/openid-configuration`: This is a MUST for OIDC compliance.
- `/openid/v1/jwks`: This is configurable through the first endpoint as you'll see later.

These endpoints provide the information of the OIDC provider and the public keys used to sign the JWT tokens, respectively. The former will be used by the service provider to validate the OIDC provider and the latter will be used to validate the JWT access tokens provided by the entities that want to talk to the Serivce Provider.

> Service Provider refers to the host that provides the service. In our example, AWS is the service provider.


Exposing such endpoints will make our OIDC provider compliant with the OIDC specification. In that regard, any OIDC compliant service provider will be able to trust our OIDC provider.

> For an OIDC provider and a Service Provider to trust each other, they must both be OIDC compliant. This means that the OIDC provider must expose certain endpoints and the Service Provider must be able to validate the OIDC provider through those endpoints.


In practice, we will need the following two absolute URLs to be accessible publicly through internet with a verified TLS certificate signed by a trusted Certificate Authority (CA):

- `https://mydomain.com/.well-known/openid-configuration`
- `https://mydomain.com/openid/v1/jwks`

Again, and just to reiterate, as per the OIDC specification the HTTPS is a must and the TLS certificate has to be signed by a trusted Certificate Authority (CA).

When all this is set up, we shall be able to add the `https://mydomain.com` to the AWS as an OIDC provider.

## Step 0: Directory Structure

There are a lot of codes we will cover in this post. It is good to know that to expect. Here's the layout of the directories we will be working with:

```plaintext
.
├── ansible.cfg
├── app/
├── configure-oidc/
├── inventory/
├── k8s/
├── playbook.yml
├── provision-k8s/
├── requirements.yml
└── vars/
```

## Step 1: Dedicated Domain Name

As mentioned, we need to assign a dedicated domain name to the OIDC provider. This will be the address we will add to the AWS IAM as an Identity Provider.

Any DNS provider will do, but for our example, we're using Cloudflare.

```hcl
variable "hetzner_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

variable "cloudflare_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

variable "stack_name" {
  type    = string
  default = "k3s-cluster"
}

variable "primary_ip_datacenter" {
  type    = string
  default = "nbg1-dc3"
}

variable "root_domain" {
  type    = string
  default = "developer-friendly.blog"
}
```

```hcl
terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.46"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.30"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

provider "hcloud" {
  token = var.hetzner_api_token
}

provider "cloudflare" {
  api_token = var.cloudflare_api_token
}
```

```hcl
resource "hcloud_primary_ip" "this" {
  for_each = toset(["ipv4", "ipv6"])

  name          = "${var.stack_name}-${each.key}"
  datacenter    = var.primary_ip_datacenter
  type          = each.key
  assignee_type = "server"
  auto_delete   = false
}
```

```hcl
data "cloudflare_zone" "this" {
  name = var.root_domain
}

resource "random_uuid" "this" {}

resource "cloudflare_record" "this" {
  zone_id = data.cloudflare_zone.this.id

  name    = "${random_uuid.this.id}.${var.root_domain}"
  proxied = false
  ttl     = 60
  type    = "A"
  value   = hcloud_primary_ip.this["ipv4"].ip_address
}


resource "cloudflare_record" "this_v6" {
  zone_id = data.cloudflare_zone.this.id

  name    = "${random_uuid.this.id}.${var.root_domain}"
  proxied = false
  ttl     = 60
  type    = "AAAA"
  value   = hcloud_primary_ip.this["ipv6"].ip_address
}
```

```hcl
output "public_ip" {
  value = hcloud_primary_ip.this["ipv4"].ip_address
}

output "public_ipv6" {
  value = hcloud_primary_ip.this["ipv6"].ip_address
}
```

We would need the required access token which you can get from their respective account settings. If you want to apply the stack, you will need a [Cloudflare token](https://dash.cloudflare.com/profile/api-tokens) and a [Hetzner API token](https://docs.hetzner.com/cloud/api/getting-started/generating-api-token/).

```shell
export TF_VAR_cloudflare_api_token="PLACEHOLDER"
export TF_VAR_hetzner_api_token="PLACEHOLDER"

tofu plan -out tfplan
tofu apply tfplan
```

## Step 2: A Live Kubernetes Cluster

At this point, we should have a live Kuberntes cluster. We've already covered how to set up a [lightweight Kubernetes cluster on a Ubuntu 22.04 machine](https://developer-friendly.blog/2024/03/22/how-to-install-lightweight-kubernetes-on-ubuntu-2204/) before and so, we won't go too deep into that.

But for the sake of completeness, we'll resurface the code one more time, with some minor tweaks here and there.

```hcl
variable "hetzner_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

variable "cloudflare_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

variable "stack_name" {
  type    = string
  default = "k3s-cluster"
}

variable "primary_ip_datacenter" {
  type    = string
  default = "nbg1-dc3"
}

variable "root_domain" {
  type    = string
  default = "developer-friendly.blog"
}

variable "server_datacenter" {
  type    = string
  default = "nbg1"
}

variable "username" {
  type    = string
  default = "k8s"
}
```

```hcl
terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.46"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.30"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.4"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }
}

provider "hcloud" {
  token = var.hetzner_api_token
}

provider "cloudflare" {
  api_token = var.cloudflare_api_token
}
```

```hcl
resource "tls_private_key" "this" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P384"
}

resource "hcloud_ssh_key" "this" {
  name       = var.stack_name
  public_key = tls_private_key.this.public_key_openssh
}

resource "hcloud_server" "this" {
  name        = var.stack_name
  server_type = "cax11"
  image       = "ubuntu-22.04"
  location    = "nbg1"

  ssh_keys = [
    hcloud_ssh_key.this.id,
  ]

  public_net {
    ipv4 = hcloud_primary_ip.this["ipv4"].id
    ipv6 = hcloud_primary_ip.this["ipv6"].id
  }

  user_data = <<-EOF
    #cloud-config
    users:
      - name: ${var.username}
        groups: users, admin, adm
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        ssh_authorized_keys:
          - ${tls_private_key.this.public_key_openssh}
    packages:
      - certbot
    package_update: true
    package_upgrade: true
    runcmd:
      - sed -i -e '/^\(#\|\)PermitRootLogin/s/^.*$/PermitRootLogin no/' /etc/ssh/sshd_config
      - sed -i -e '/^\(#\|\)PasswordAuthentication/s/^.*$/PasswordAuthentication no/' /etc/ssh/sshd_config
      - sed -i '$a AllowUsers ${var.username}' /etc/ssh/sshd_config
      - |
        curl https://get.k3s.io | \
          INSTALL_K3S_VERSION="v1.29.3+k3s1" \
          INSTALL_K3S_EXEC="--disable traefik
            --kube-apiserver-arg=service-account-jwks-uri=https://${cloudflare_record.this.name}/openid/v1/jwks
            --kube-apiserver-arg=service-account-issuer=https://${cloudflare_record.this.name}
            --disable-network-policy
            --flannel-backend none
            --write-kubeconfig /home/${var.username}/.kube/config
            --secrets-encryption" \
          sh -
      - chown -R ${var.username}:${var.username} /home/${var.username}/.kube/
      - |
        CILIUM_CLI_VERSION=v0.16.4
        CLI_ARCH=arm64
        curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/$CILIUM_CLI_VERSION/cilium-linux-$CLI_ARCH.tar.gz{,.sha256sum}
        sha256sum --check cilium-linux-$CLI_ARCH.tar.gz.sha256sum
        sudo tar xzvfC cilium-linux-$CLI_ARCH.tar.gz /usr/local/bin
      - kubectl completion bash | tee /etc/bash_completion.d/kubectl
      - k3s completion bash | tee /etc/bash_completion.d/k3s
      - |
        cat << 'EOF2' >> /home/${var.username}/.bashrc
        alias k=kubectl
        complete -F __start_kubectl k
        EOF2
      - reboot
  EOF
}
```

```hcl
data "http" "this" {
  url = "https://checkip.amazonaws.com"
}

resource "hcloud_firewall" "this" {
  name = var.stack_name

  rule {
    direction   = "in"
    protocol    = "tcp"
    port        = 22
    source_ips  = [format("%s/32", trimspace(data.http.this.response_body))]
    description = "Allow SSH access from my IP"
  }

  rule {
    direction = "in"
    protocol  = "tcp"
    port      = 80
    source_ips = [
      "0.0.0.0/0",
      "::/0",
    ]
    description = "Allow HTTP access from everywhere"
  }

  rule {
    direction = "in"
    protocol  = "tcp"
    port      = 443
    source_ips = [
      "0.0.0.0/0",
      "::/0",
    ]
    description = "Allow HTTPS access from everywhere"
  }


  depends_on = [
    hcloud_server.this,
  ]
}

resource "hcloud_firewall_attachment" "this" {
  firewall_id = hcloud_firewall.this.id
  server_ids  = [hcloud_server.this.id]
}
```

```hcl
output "public_ip" {
  value = hcloud_primary_ip.this["ipv4"].ip_address
}

output "public_ipv6" {
  value = hcloud_primary_ip.this["ipv6"].ip_address
}

output "ssh_private_key" {
  value     = tls_private_key.this.private_key_pem
  sensitive = true
}

output "ansible_inventory_yaml" {
  value = <<-EOF
    k8s:
      hosts:
        ${var.stack_name}:
          ansible_host: ${hcloud_server.this.ipv4_address}
          ansible_user: ${var.username}
          ansible_ssh_private_key_file: ~/.ssh/k3s-cluster
          ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no'
  EOF
}

output "ansible_vars" {
  value = <<-EOF
    domain_name: ${cloudflare_record.this.name}
  EOF
}

output "oidc_provider_url" {
  value = cloudflare_record.this.name
}
```

Notice the lines where we specify the OIDC issuer URL & JWK URL for the Kubernetes API server to be a publicly accessible address and pass it as an argument to the `k3s` server.

```shell
--kube-apiserver-arg=service-account-jwks-uri=https://${cloudflare_record.this.name}/openid/v1/jwks
--kube-apiserver-arg=service-account-issuer=https://${cloudflare_record.this.name}
```

If not specified, the rest of this guide won't work and additional configuration is required. In summary, these are the URLs that will be used by the Service Provider when trying to verify the OIDC provider & the access tokens of the Service Accounts.

Business as usual, we apply the stack as below.

```shell
tofu plan -out tfplan
tofu apply tfplan
```

And for connecting to the machine:

```shell
tofu output -raw ssh_private_key > ~/.ssh/k3s-cluster
chmod 600 ~/.ssh/k3s-cluster

IP_ADDRESS=$(tofu output -raw public_ip)
ssh -i ~/.ssh/k3s-cluster k8s@$IP_ADDRESS
```

To be able to use the Ansible playbook in the next steps, we shall write the inventory where Ansible expects them.

```ini
# ansible.cfg
[defaults]
become = false
cache_timeout = 3600
fact_caching = ansible.builtin.jsonfile
fact_caching_connection = /tmp/ansible_facts
gather_facts = false
interpreter_python = auto_silent
inventory = ./inventory
log_path = /tmp/ansible.log
roles_path = ~/.ansible/roles:./roles
ssh_common_args = -o ConnectTimeout=5
verbosity = 2

[inventory]
cache = true
cache_connection = /tmp/ansible_inventory
enable_plugins = 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml', 'azure_rm', 'aws_ec2', 'auto'
```

```shell
mkdir -p ../inventory/group_vars
tofu output -raw ansible_inventory_yaml > ../inventory/k3s-cluster.yml
tofu output -raw ansible_vars > ../inventory/group_vars/all.yml
```

The result of `ansible-inventory --list`:

```json
{
  "_meta": {
    "hostvars": {
      "k3s-cluster": {
        "ansible_host": "XX.XX.XX.XX",
        "ansible_ssh_common_args": "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no",
        "ansible_ssh_private_key_file": "~/.ssh/k3s-cluster",
        "ansible_user": "k8s",
        "discovered_interpreter_python": {
          "__ansible_unsafe": "/usr/bin/python3"
        }
      }
    }
  },
  "all": {
    "children": [
      "ungrouped",
      "k8s"
    ]
  },
  "k8s": {
    "hosts": [
      "k3s-cluster"
    ]
  }
}
```

At this stage we're ready to move on to the next step.

## Step 3: Bootstrap the Cluster

At this point we have installed the Cilium binary in our host machine, yet we haven't installed the CNI plugin in our Kubernetes cluster.

Let's create an Ansible role and a playbook to take care of all the Day 1 operations.

```shell
ansible-galaxy init k8s
touch playbook.yml
```

The first step is to install the Cilium CNI.

```yaml
# k8s/defaults/main.yml
---
cilium_version: 1.15.4
```

```yaml
# k8s/tasks/cilium.yml
---
- name: Install cilium
  ansible.builtin.command:
    cmd: cilium install --set kubeProxyReplacement=true --wait --version {{ cilium_version }}
  register: cilium_install
  changed_when: false
  ignore_errors: true
  environment:
    KUBECONFIG: /etc/rancher/k3s/k3s.yaml
```

```yaml
# k8s/tasks/main.yml
---
- name: Cilium
  include_tasks: cilium.yml
  tags: cilium
```

```yaml
# playbook.yml
---
- name: Bootstrap k8s node
  hosts: k3s-cluster
  gather_facts: false
  become: true
  roles:
    - k8s
```


To run the playbook:

```shell
ansible-playbook playbook.yml
```

## Step 4: Fetch the TLS Certificate

At this point, we need a CA verified TLS certificate for the domain name we created in the first step.

We will carry our tasks with Ansible throughout the entire Day 1 to Day n operations.

```yaml
# k8s/defaults/main.yml
---
cilium_version: 1.15.4
acme_home: /var/www/html
```

```ini
# k8s/templates/wellknown-server.service.j2
[Unit]
Description=Wellknown Server

[Service]
ExecStartPre=/bin/mkdir -p {{ acme_home }}/.well-known/acme-challenge
ExecStart=/usr/bin/python3 -m http.server -d {{ acme_home }} 80
Restart=on-failure
WorkingDirectory={{ acme_home }}
User=acme
Group=acme
RestartSec=5
AmbientCapabilities=cap_net_bind_service

[Install]
WantedBy=multi-user.target
```

```yaml
# k8s/handlers/main.yml
---
- name: Restart wellknown-server
  ansible.builtin.systemd:
    name: wellknown-server
    state: restarted
    daemon_reload: true
```

```yaml
# k8s/tasks/certbot.yml
---
- name: Create acme group
  ansible.builtin.group:
    name: acme
    state: present
    system: true
- name: Create acme user
  ansible.builtin.user:
    name: acme
    state: present
    group: acme
    shell: /bin/false
    system: true
    create_home: false
- name: Create working dir for acme user
  ansible.builtin.file:
    path: "{{ acme_home }}"
    state: directory
    owner: acme
    group: acme
    mode: "0755"
- name: Create an standalone server to respond to challenges
  ansible.builtin.template:
    src: wellknown-server.service.j2
    dest: /etc/systemd/system/wellknown-server.service
    owner: root
    group: root
    mode: "0644"
  notify: Restart wellknown-server
- name: Start the wellknown-server
  ansible.builtin.systemd:
    name: wellknown-server
    state: started
    enabled: true
    daemon_reload: true
- name: Use certbot to fetch TLS certificate for {{ domain_name }}
  ansible.builtin.command:
    cmd: >-
      certbot certonly
        --webroot
        -w {{ acme_home }}
        --non-interactive
        --agree-tos
        --email {{ domain_email }}
        --domains {{ domain_name }}
  args:
    creates: /etc/letsencrypt/live/{{ domain_name }}/fullchain.pem
```

```yaml
# k8s/tasks/main.yml
---
- name: Cilium
  ansible.builtin.import_tasks: cilium.yml
  tags:
    - cilium
- name: Certbot
  ansible.builtin.import_tasks: certbot.yml
  tags:
    - certbot
```

```yaml
# playbook.yml
---
- name: Bootstrap k8s node
  hosts: k3s-cluster
  gather_facts: false
  become: true
  vars:
    domain_email: admin@developer-friendly.blog
  roles:
    - k8s
```

### Certificate Renewal


Although not required, one of the benefits of using `certbot` for
TLS certificates is the ease of renewal.

After your initial `certbot` command, you will find the following two
`systemd` files in your system.

```ini
# /lib/systemd/system/certbot.service
[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://certbot.eff.org/docs
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true
```

```ini
# /lib/systemd/system/certbot.timer
[Unit]
Description=Run certbot twice daily

[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true

[Install]
WantedBy=timers.target
```

Although on the same host, you will find a crontab entry for the `certbot`
as you see below:

```plaintext
# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
#
# Important Note!  This cronjob will NOT be executed if you are
# running systemd as your init system.  If you are running systemd,
# the cronjob.timer function takes precedence over this cronjob.  For
# more details, see the systemd.timer manpage, or use systemctl show
# certbot.timer.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew
```

All of these files are created by the `certbot` binary during the initial
run. You are free to modify and customize it, although it's unlikely that
you will need to.

After adding another task to our Ansible role, we can run the new tasks with the following command:

```shell
ansible-playbook playbook.yml --tags certbot
```

## Step 5: Expose OIDC Configuration to the Internet

We've prepared all these works so far for this next step.

In here, we will fetch the OIDC configuration from the Kubernetes API server and expose them to the internet on HTTPS using the newly acquired TLS certificate with the help of [static web server](https://static-web-server.net/).

```yaml
# k8s/defaults/main.yml
---
cilium_version: 1.15.4
acme_home: /var/www/html
static_web_server_home: /var/www/static-web-server
kubeconfig: /etc/rancher/k3s/k3s.yaml
```

```yaml
# k8s/handlers/main.yml
---
- name: Restart wellknown-server
  ansible.builtin.systemd:
    name: wellknown-server
    state: restarted
    daemon_reload: true
- name: Restart static-web-server-prepare
  ansible.builtin.systemd:
    name: static-web-server-prepare
    state: restarted
    daemon_reload: true
- name: Restart static-web-server
  ansible.builtin.systemd:
    name: static-web-server
    state: restarted
    daemon_reload: true
```

```ini
# k8s/templates/static-web-server.service.j2
[Unit]
Description=Static Web Server

[Service]
ExecStartPre=/usr/bin/test -s cert.pem
ExecStartPre=/usr/bin/test -s key.pem
ExecStartPre=/usr/bin/test -s .well-known/openid-configuration
ExecStartPre=/usr/bin/test -s openid/v1/jwks
ExecStart=/usr/local/bin/static-web-server \
  --host 0.0.0.0 \
  --port 443 \
  --root . \
  --log-level info \
  --http2 \
  --http2-tls-cert cert.pem \
  --http2-tls-key key.pem \
  --compression \
  --health \
  --experimental-metrics
Restart=on-failure
WorkingDirectory={{ static_web_server_home }}
User=static-web-server
Group=static-web-server
RestartSec=5
AmbientCapabilities=cap_net_bind_service

[Install]
WantedBy=multi-user.target
```

```yaml
# k8s/tasks/static-server.yml
- name: Create static-web-server group
  ansible.builtin.group:
    name: static-web-server
    state: present
    system: true
- name: Create static-web-server user
  ansible.builtin.user:
    name: static-web-server
    state: present
    group: static-web-server
    shell: /bin/false
    system: true
    create_home: false
- name: Create working dir for static-web-server user
  ansible.builtin.file:
    path: "{{ static_web_server_home }}"
    state: directory
    owner: static-web-server
    group: static-web-server
    mode: "0755"
- name: Download static web server binary
  ansible.builtin.get_url:
    url: "{{ static_web_server_download_url }}"
    dest: "/tmp/{{ static_web_server_download_url | basename }}"
    checksum: "sha256:{{ static_web_server_checksum }}"
    owner: root
    group: root
    mode: "0644"
  register: download_static_web_server
- name: Extract static web server binary
  ansible.builtin.unarchive:
    src: "{{ download_static_web_server.dest }}"
    dest: /usr/local/bin/
    owner: root
    group: root
    mode: "0755"
    remote_src: true
    extra_opts:
      - --strip-components=1
      - --wildcards
      - "**/static-web-server"
  notify: Restart static-web-server
- name: Create static-web-server-prepare script
  ansible.builtin.template:
    src: static-web-server-prepare.sh.j2
    dest: /usr/local/bin/static-web-server-prepare
    owner: root
    group: root
    mode: "0755"
  notify: Restart static-web-server-prepare
- name: Create static-web-server-prepare service
  ansible.builtin.template:
    src: static-web-server-prepare.service.j2
    dest: /etc/systemd/system/static-web-server-prepare.service
    owner: root
    group: root
    mode: "0644"
  notify: Restart static-web-server-prepare
- name: Create static-web-server-prepare timer
  ansible.builtin.template:
    src: static-web-server-prepare.timer.j2
    dest: /etc/systemd/system/static-web-server-prepare.timer
    owner: root
    group: root
    mode: "0644"
  notify: Restart static-web-server-prepare
- name: Start static-web-server-prepare
  ansible.builtin.systemd:
    name: static-web-server-prepare.timer
    state: started
    enabled: true
    daemon_reload: true
- name: Create static-web-server service
  ansible.builtin.template:
    src: static-web-server.service.j2
    dest: /etc/systemd/system/static-web-server.service
    owner: root
    group: root
    mode: "0644"
  notify: Restart static-web-server
- name: Start static-web-server service
  ansible.builtin.systemd:
    name: static-web-server
    state: started
    enabled: true
    daemon_reload: true
```

```yaml
# k8s/tasks/main.yml
---
- name: Cilium
  ansible.builtin.import_tasks: cilium.yml
  tags:
    - cilium
- name: Certbot
  ansible.builtin.import_tasks: certbot.yml
  tags:
    - certbot
- name: Static web server
  ansible.builtin.import_tasks: static-server.yml
  tags:
    - static-web-server
```

```yaml
# vars/aarch64.yml
---
static_web_server_download_url: https://github.com/static-web-server/static-web-server/releases/download/v2.28.0/static-web-server-v2.28.0-armv7-unknown-linux-musleabihf.tar.gz
static_web_server_checksum: 492dda3749af5083e5387d47573b43278083ce62de09b2699902e1ba40bf1e45
```

```yaml
# playbook.yml
---
- name: Bootstrap k8s node
  hosts: k3s-cluster
  gather_facts: true
  become: true
  vars:
    domain_email: admin@developer-friendly.blog
  vars_files:
    - vars/{{ ansible_architecture }}.yml
  roles:
    - k8s
  tags:
    - provision
```

Running this will be as follows:

```shell
ansible-playbook playbook.yml --tags static-web-server
```

You can notice that we have turned on fact gathering in this step. This is due to our desire to include host-specific variables as you see with `vars_files` entry.

From the above tasks, there are references to a couple of important files. One is the `static-web-server-prepare` which has both a `service` file as well as a `timer` file.

This gives us flexibility to define `oneshot` services which will only run to completion on every tick of the `timer`. Effectively, we'll be able to separate the executable task and the scheduling of the task.

The definitions for those files are as following:

```shell
# k8s/templates/static-web-server-prepare.sh.j2
#!/usr/bin/env sh

# This script will run as root to prepare the files for the static web server

set -eu

mkdir -p {{ static_web_server_home }}/.well-known \
  {{ static_web_server_home }}/openid/v1

kubectl get --raw /.well-known/openid-configuration > \
  {{ static_web_server_home }}/.well-known/openid-configuration

kubectl get --raw /openid/v1/jwks > \
  {{ static_web_server_home }}/openid/v1/jwks

cp /etc/letsencrypt/live/{{ domain_name }}/fullchain.pem \
  {{ static_web_server_home }}/cert.pem

cp /etc/letsencrypt/live/{{ domain_name }}/privkey.pem \
  {{ static_web_server_home }}/key.pem


chown -R static-web-server:static-web-server {{ static_web_server_home }}
```

Notice how we are manually fetching the OIDC configurations from the Kubernetes as well as the TLS certificate. This is due to a possibility of renewal for any of the given files:

1. Firstly, the Kubernetes API server might rotate its Service Account issuer key pair and with that, the JWKs URL will have different output.
2. Secondly, the TLS certificate will be renewed by `certbot` in the background and we have to keep up with that.

Now, let's take a look at our preparation service and timer definition.

```ini
# k8s/templates/static-web-server-prepare.service.j2
[Unit]
Description=Update TLS & K8s OIDC Config

[Service]
Environment=KUBECONFIG={{ kubeconfig }}
ExecStartPre=/bin/mkdir -p .well-known openid/v1
ExecStart=/usr/local/bin/static-web-server-prepare
Restart=on-failure
Type=oneshot
WorkingDirectory={{ static_web_server_home }}

[Install]
WantedBy=multi-user.target
```

```ini
# k8s/templates/static-web-server-prepare.timer.j2
[Unit]
Description=Update TLS & K8s OIDC Config Every Minute

[Timer]
OnCalendar=*-*-* *:*:00

[Install]
WantedBy=multi-user.target
```

Notice that the service file specifies the working directory for the script. Which means the `static-web-server-prepare` shell script will be executed in the specified directory.

Also, watch out for `oneshot` systemd service type. These services are not long-running processes in an infitie loop. Instead, they will run to completion and the systemd will not report their state as `Active` as it would with [`simple` services](https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html).

## Step 6: Add the OIDC Provider to AWS

That's it. We have done all the hard work. Anything after this will be a breeze compared to what we've done so far as you shall see shortly.

Now, we have a domain name that is publishing its OIDC configuration and JWKs over the HTTPS endpoint and is ready to be used as a trusted OIDC provider.

All we need right now, is a couple of TF resource in the AWS account and after that, we can test the setup using a sample Job that takes a Service Account in its definition and uses its token to talk to AWS.

Note that we're starting a new TF module below.

```hcl
terraform {
  required_providers {
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.46"
    }
  }
}
```

```hcl
data "terraform_remote_state" "k8s" {
  backend = "local"

  config = {
    path = "../provision-k8s/terraform.tfstate"
  }
}
```

```hcl
data "tls_certificate" "this" {
  url = "https://${data.terraform_remote_state.k8s.outputs.oidc_provider_url}"
}

resource "aws_iam_openid_connect_provider" "this" {
  url = "https://${data.terraform_remote_state.k8s.outputs.oidc_provider_url}"

  # JWT token audience (aud)
  client_id_list = [
    "sts.amazonaws.com"
  ]

  thumbprint_list = [
    data.tls_certificate.this.certificates[0].sha1_fingerprint
  ]
}
```

Let's apply this stack:

```shell
export AWS_PROFILE="PLACEHOLDER"

tofu plan -out tfplan
tofu apply tfplan
```

Believe it or not, but after all these efforts, it is finally done.

Now it is time for the test.

In order to be able to assume a role from inside a pod of our cluster, we will create a sample IAM Role with a trust relationship to the OIDC provider we just created.

```hcl
data "aws_iam_policy_document" "this" {
  statement {
    actions = [
      "sts:AssumeRoleWithWebIdentity"
    ]

    effect = "Allow"

    principals {
      type = "Federated"
      identifiers = [
        aws_iam_openid_connect_provider.this.arn
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "${aws_iam_openid_connect_provider.this.url}:aud"

      values = [
        "sts.amazonaws.com"
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "${aws_iam_openid_connect_provider.this.url}:sub"

      values = [
        "system:serviceaccount:${var.service_account_namespace}:${var.service_account_name}"
      ]
    }
  }
}

resource "aws_iam_role" "this" {
  name               = "k3s-demo-app"
  assume_role_policy = data.aws_iam_policy_document.this.json
  managed_policy_arns = [
    "arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess"
  ]
}
```

```hcl
output "iam_role_arn" {
  value = aws_iam_role.this.arn
}

output "service_account_namespace" {
  value = var.service_account_namespace
}

output "service_account_name" {
  value = var.service_account_name
}
```

The AWS IAM Role trust relationship will look something like this:

```json
{
  "Statement": [
    {
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "4f0fce7c-9efa-9ee3-5fe0-467d95d2584c.developer-friendly.blog:aud": "sts.amazonaws.com",
          "4f0fce7c-9efa-9ee3-5fe0-467d95d2584c.developer-friendly.blog:sub": "system:serviceaccount:default:demo-service-account"
        }
      },
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/4f0fce7c-9efa-9ee3-5fe0-467d95d2584c.developer-friendly.blog"
      }
    }
  ],
  "Version": "2012-10-17"
}
```

This, of course, shouldn't come as a surprise. We have already seen this in the TF definition above.

## Step 7: Test the Setup

We have created the IAM Role with the trust relationship to the OIDC provider of the cluster. With the conditional in the AWS IAM Role you se in the previous step, only the Service Accounts with the specified audience, in the `default`namespace and with the Service Account name `demo-service-account` will be able to assume the role.

That said, let's use create another Ansible role to create a Kuberentes Job.

```shell
ansible-galaxy init app
```

We will need the Kubernetes core Ansible collection, so let's install that.

```yaml
# requirements.yml
- name: kubernetes.core
  version: 2.4.1
```

```shell
ansible-galaxy collection install -r requirements.yml
```

```yaml
# app/defaults/main.yml
---
aws_region: eu-central-1
role_session_name: k3s-cluster
```

```yaml
# app/templates/manifest.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-service-account
  namespace: default
---
apiVersion: batch/v1
kind: Job
metadata:
  name: demo-app
  namespace: default
spec:
  selector:
    matchLabels:
      job-name: demo-app
  template:
    metadata:
      labels:
        job-name: demo-app
    spec:
      restartPolicy: Never
      containers:
        - image: amazon/aws-cli:2.15.40
          name: demo-app
          command:
            - sh
            - -c
            - |
              aws sts get-caller-identity

              aws ssm get-parameters-by-path \
                --path / --recursive \
                --with-decryption \
                --query "Parameters[*].[Name]" \
                --output text
          env:
            - name: AWS_REGION
              value: "{{ aws_region }}"
            - name: AWS_ROLE_ARN
              value: "{{ role_arn }}"
            - name: AWS_ROLE_SESSION_NAME
              value: "{{ role_session_name }}"
            - name: AWS_WEB_IDENTITY_TOKEN_FILE
              value: /var/run/secrets/tokens/token
          securityContext:
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: token
              mountPath: /var/run/secrets/tokens
              readOnly: true
            - name: aws-config
              mountPath: /root/.aws
      serviceAccountName: demo-service-account
      volumes:
        - name: token
          projected:
            sources:
              - serviceAccountToken:
                  path: token
                  audience: sts.amazonaws.com
        - name: aws-config
          emptyDir: {}
```

```yaml
# app/tasks/main.yml
---
- name: Apply the app job
  kubernetes.core.k8s:
    template: manifest.yml
    state: present
    force: true
    wait: true
```

```yaml
# playbook.yml
---
- name: Bootstrap k8s node
  hosts: k3s-cluster
  gather_facts: true
  become: true
  vars:
    domain_email: admin@developer-friendly.blog
  vars_files:
    - vars/{{ ansible_architecture }}.yml
  roles:
    - k8s
  tags:
    - provision
- name: Test the AWS Access
  hosts: k3s-cluster
  gather_facts: false
  become: true
  environment:
    KUBECONFIG: /etc/rancher/k3s/k3s.yaml
  pre_tasks:
    - name: Install pip3
      ansible.builtin.package:
        name: python3-pip
        state: present
    - name: Install kubernetes library
      ansible.builtin.pip:
        name: kubernetes<30
        state: present
    - name: Read Tofu output from ./configure-oidc
      ansible.builtin.command:
        cmd: tofu output -raw iam_role_arn
        chdir: "{{ playbook_dir }}/configure-oidc"
      delegate_to: localhost
      become: false
      changed_when: false
      register: configure_oidc
    - name: Set the AWS role arn
      ansible.builtin.set_fact:
        role_arn: "{{ configure_oidc.stdout }}"
  roles:
    - app
  tags:
    - test
    - never
```

A few important notes are worth mentioning here:

1. The second playbook is tagged with `never`. That is because there is a dependency on the second TF module. We have to manually resolve it before being able to run the second playbook. As soon as the dependency is resolved, we can run the second playbook with the `--tags test` flag.
2. There is a fact gathering in the `pre_task` of the second playbook. That is, again, because of the dependency to the TF module. We will grab the output of the TF module and pass it to our next role. If you notice there is a `aws_region` variable in the Jinja template that is being initialized by this fact gathering step.
3. In the fact gathering step, there is an Ansible delegation happening. This will ensure that the task is running in our own machine and not the target machine. The reason is that the TF module and its TF state file is in our local machine. We also do not need the `become` and as such it is turned off.
4. You will notice that the job manifest is using AWS CLI Docker image. By specifying some of the [expected environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html), we are able to use the AWS CLI without the requirement of manual `aws configure`.

This playbook can be run after the second TF module with the following command:

```shell
ansible-playbook playbook.yml --tags test
```

When checking the logs of the deployed Kubernetes Job, we can see that it has been successful.

```shell
kubectl logs job/demp-app
```

There is no AWS SSM Parameter in the target AWS account and as such, the
AWS CLI will not return an empty list; it will return nothing!

```json
{
  "Account": "XXXXXXXXXXXX",
  "Arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/k3s-demo-app/k3s-cluster",
  "UserId": "AROAYTLV5GLUYXO2EKETN:k3s-cluster"
}
```

Lastly, to test if the Service Account and the IAM Role trust policy plays any role in any of this, we can remove the `serviceAccountToken` and try to recreate the job.

The output is as expected:

```plaintext
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
```

That's all folks! We can now wrap this up.

## Bonus: JWKs URL

Remember at the beginning of this guide when we mentioned that the JWKs URL is configurable through the OIDC configuration endpoint?

Let's see it in action.

```shell
DOMAIN=$(grep domain_name inventory/group_vars/all.yml | awk '{print $2}')
curl https://$DOMAIN/.well-known/openid-configuration | jq -r .jwks_uri
```

This means that you can host your JWKs on a different server than the OIDC server. Although I don't suggest this to be a good idea because of all the maintenance overhead.

That said, if your JWKs URL is at a different server or hosted on a different endpoint, all you gotta do, is pass the value to the `kube-apiserver` as you see below:

```shell
kube-apiserver --service-account-jwks-uri=https://mydomain.com/some/different/endpoint
```

## Conclusion

OpenID Connect is one of the most powerful protocols that powers the internet. Yet, it is so underestimated and easily overlooked. If you look closely enough on any system around you, you will see a lot of practical applications of OIDC.

One of the cues that you can look for when trying to identify applicability of OIDC is when trying to authenticate an identity of one system to another. You will almost always never need to create another identity in the target system, nor do you need to pass any credentials around. All that's needed is to establish a trust relationship between the two systems and you're good to go.

This gives you a lot of flexibility and enhances your security posture. You will also remove the overhead of secret rotations from your workload.

In this post, we have seen how to establish a trust relationship between a bear-metal Kubernetes cluster and AWS IAM to grant cluster generated Service Account tokens access to AWS services using OIDC.

Having this foundation in place, it's easy to extend this pattern to managed Kubernetes clusters such as [Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/) or [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/docs). All you need from the managed Kubernetes cluster is the OIDC configuration endpoint, which in turn has the JWKs URL. With that, you can create the trust relationship in AWS or any other Service Provider and grant the relevant access to your services as needed.

Hope you've enjoyed reading the post as much as I've enjoyed writing it. I wish you have learned something new and useful from it.

Until next time, _ciao_ 🤠 & happy coding! :penguin: :crab: