AWS: Karpenter and SSH for Kubernetes WorkerNodes
Tags: security, aws, devops, kubernetes
Published: 2024-06-23 14:36:36

---
title: AWS: Karpenter and SSH for Kubernetes WorkerNodes
published: true
tags: security,aws,devops,kubernetes
description: Setting up SSH access to EC2 created by Karpenter in EKS from AWS Session Manager, AWS EC2 Instance Connect, and via EC2 User Data manually and from Terraform
canonical_url: https://rtfm.co.ua/en/aws-karpenter-and-ssh-for-kubernetes-workernodes/
---

![](https://cdn-images-1.medium.com/max/1024/1*JZAYCCIy6Muaxtz2FKpu0w.png)

We have an AWS EKS cluster with WorkerNodes/EC2 created with Karpenter.

The process of creating the infrastructure, cluster, and launching Karpenter is described in previous posts:

- [Terraform: Building EKS, part 1 — VPC, Subnets and Endpoints](https://rtfm.co.ua/en/terraform-building-eks-part-1-vpc-subnets-and-endpoints/)
- [Terraform: Building EKS, part 2 — an EKS cluster, WorkerNodes, and IAM](https://rtfm.co.ua/en/terraform-building-eks-part-2-an-eks-cluster-workernodes-and-iam/)
- [Terraform: Building EKS, part 3 — Karpenter installation](https://rtfm.co.ua/en/terraform-building-eks-part-3-karpenter-installation/)

What this system really lacks, is access to servers via SSH, without which you feel like… Well, like DevOps, not Infrastructure Engineer. In short, SSH access is sometimes necessary, but — surprise — Karpenter does not allow you to add a key to the WorkerNodes it manages out of the box.

Although, what’s the problem to add in the [`EC2NodeClass`](https://karpenter.sh/v0.32/concepts/nodeclasses/) a way to pass an SSH key, as it is done in the Terraform's `resource "aws_instance`" with the parameter `key_name?`

But it’s okay. If it’s not there, it’s not there. Maybe they will add it later.

Instead, Karpenter’s documentation [Can I add SSH keys to a NodePool?](https://karpenter.sh/docs/faq/#can-i-add-ssh-keys-to-a-nodepool) suggests using either [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) or [AWS EC2 Instance Connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-methods.html), or “the old school way” — add the public part of the key via [AWS EC2 User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html), and connect via a bastion host or a VPN.

So what we’re going to do today is:

- try all three solutions one by one
- first we’ll do each one by hand, then we’ll see how to add it to our automation with Terraform
- and then we’ll decide which option will be the easiest

### Option 1: AWS Systems Manager Session Manager and SSH on EC2

AWS Systems Manager Session Manager is used to manage EC2 instances. In general, it can do quite a lot, for example, keep track of patches and updates for packages that are installed on instances.

Currently, we are only interested in it as a system that will allow us to have an SSH to a Kubernetes WorkerNode.

It requires an SSM agent, which is installed by default on all instances with Amazon Linux AMI.

Find nodes created by Karpenter (we have a dedicated label for them):

```
$ kubectl get node -l created-by=karpenter
NAME STATUS ROLES AGE VERSION
ip-10-0-34-239.ec2.internal Ready <none> 21h v1.28.8-eks-ae9a62a
ip-10-0-35-100.ec2.internal Ready <none> 9m28s v1.28.8-eks-ae9a62a
ip-10-0-39-0.ec2.internal Ready <none> 78m v1.28.8-eks-ae9a62a
...
```

Find an Instance ID:

```
$ kubectl get node ip-10-0-34-239.ec2.internal -o json | jq -r ".spec.providerID" | cut -d \/ -f5 
i-011b1c0b5857b0d92
```

### AWS CLI: TargetNotConnected when calling the StartSession operation

Try to connect and get the error “ **TargetNotConnected** ”:

```
$ aws --profile work ssm start-session --target i-011b1c0b5857b0d92

An error occurred (TargetNotConnected) when calling the StartSession operation: i-011b1c0b5857b0d92 is not connected.
```

Or through the AWS Console:

![](https://cdn-images-1.medium.com/max/1024/0*pd-hoF-_KGH3OUdi.png)

But here, too, we have the connection error — “ **SSM Agent is not online** ”:

![](https://cdn-images-1.medium.com/max/857/0*XyxBBm3BPzT-tWGa.png)

The error occurs because:

- either the IAM role that is connected to the instance does not have SSM permissions
- or EC2 is running on a private network and the agent cannot connect to an external endpoint

### SessionManager and IAM Policy

Let’s check. Find the IAM Role attached to this EC2:

![](https://cdn-images-1.medium.com/max/522/0*b0om-9XRf8Y6PT3A.png)

And the policies connected to it — there is nothing about SSM:

![](https://cdn-images-1.medium.com/max/459/0*CcTxN7RjXo6QLdvi.png)

Edit the Role by hand for now, then we will do it in Terraform code:

![](https://cdn-images-1.medium.com/max/341/0*1sIR_EKDzboZ3Blm.png)

Attach the `AmazonSSMManagedInstanceCore`:

![](https://cdn-images-1.medium.com/max/802/0*OzhS2NZ-9HiDcX4N.png)

And in a minute or two, try again:

![](https://cdn-images-1.medium.com/max/875/0*5LOuJamPFjn4UPZ2.png)

![](https://cdn-images-1.medium.com/max/876/0*13njIztTj-jrixDB.png)

### SessionManager and VPC Endpoint

Another possible reason for the problems connecting the SSM agent to AWS is that the instance does not have access to SSM endpoints:

- ssm.region.amazonaws.com
- ssmmessages.region.amazonaws.com
- ec2messages.region.amazonaws.com

If the subnet is private, and has limits on external connections, then you may need to create a [VPC Endpoint](https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html) for SSM.

See [SSM Agent is not online](https://repost.aws/questions/QUCuCl8hlTR_Sdnn7GrGsr4w/ssm-agent-is-not-online-the-ssm-agent-was-unable-to-connect-to-a-systems-manager-endpoint-to-register-itself-with-the-service) and [Troubleshooting Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-troubleshooting.html).

### AWS CLI: SessionManagerPlugin is not found

However, after the IAM fix, when connecting from a workstation using the AWS CLI, we can get the “ **SessionManagerPlugin is not found** ” error:

```
$ aws --profile work ssm start-session --target i-011b1c0b5857b0d92
SessionManagerPlugin is not found. Please refer to SessionManager Documentation here: http://docs.aws.amazon.com/console/systems-manager/session-manager-plugin-not-found
```

Install it locally — see the documentation [Installing the Session Manager Plugin for the AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html).

For Arch Linux there is a [`aws-session-manager-plugin`](https://aur.archlinux.org/packages/aws-session-manager-plugin) package in AUR:

```
$ yay -S aws-session-manager-plugin
```

And now we can connect:

```
$ aws --profile work ssm start-session --target i-011b1c0b5857b0d92
Starting session with SessionId: arseny-33ahofrlx7bwlecul2mkvq46gy
sh-4.2$
```

All that’s left is to add it to the automation.

### Terraform: EKS module, and adding an IAM Policy

For the Terraform EKS module from Anton Babenko, we can add a policy through the `iam_role_additional_policies` parameter - see the [`node_groups.tf`](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/7cd3be3fbbb695105a447b37c4653a00b0b51b94/node_groups.tf#L395), and in the examples of the [AWS EKS Terraform module](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/18.16.0#usage).

In the 20.0 module, the parameter name has changed — `iam_role_additional_policies` => `node_iam_role_additional_policies`, but we are still using version 19.21.0, and the role is added in this way:

```
...
module "eks" {
  source = "terraform-aws-modules/eks/aws"
  version = "~> 19.21.0"
  cluster_name = local.env_name
  cluster_version = var.eks_version
  ...
  vpc_id = local.vpc_out.vpc_id
  subnet_ids = data.aws_subnets.private.ids
  control_plane_subnet_ids = data.aws_subnets.intra.ids
  manage_aws_auth_configmap = true

  eks_managed_node_groups = {
      ...
      # allow SSM
      iam_role_additional_policies = {
        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }
...
```

Remove what we did manually, deploy the Terraform code, and check that the Policy has been added:

![](https://cdn-images-1.medium.com/max/492/0*nyZ1-8L-1SUELRCO.png)

And the connection is working:

```
$ aws --profile work ssm start-session --target i-011b1c0b5857b0d92
Starting session with SessionId: arseny-pt7d44xp6ibvqcezj2oqjaxv5q
sh-4.2$ bash
[ssm-user@ip-10-0-34-239 bin]$ pwd
/usr/bin
```

### Option 2: AWS EC2 Instance Connect and SSH on EC2

Another way to connect is through EC2 Instance Connect. Documentation — [Connect to your Linux instance with EC2 Instance Connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-eic.html).

It also requires an agent, which is also installed by default on the Amazon Linux.

For instances on private networks, EC2 Instance Connect VPC Endpoint is required for connection.

### SecurityGroup and SSH

Instance Connect through the endpoint requires access to port 22, SSH (as opposed to SSM, which opens a connection through the agent itself).

Open the port for all addresses in the VPC:

![](https://cdn-images-1.medium.com/max/1024/0*lWrdjbMiDkzM4Nzr.png)

![](https://cdn-images-1.medium.com/max/1024/0*zF-pNyZ2MlwHPNRR.png)

### EC2 Instance Connect VPC Endpoint

Go to the VPC Endpoints and create an endpoint:

![](https://cdn-images-1.medium.com/max/1024/0*85OhRfXWQgPvwYAT.png)

Select the _EC2 Instance Connect Endpoint_ type, the VPC itself, and the SecurityGroup:

![](https://cdn-images-1.medium.com/max/1024/0*-mdQWwbOGo5ztQi3.png)

Choose a Subnet — we have most of the resources in _us-east-1a_, so we’ll use it to avoid unnecessary cross-AvailabilityZone traffic (see [AWS: Cost optimization — services expenses overview and traffic costs in AWS](https://rtfm.co.ua/en/aws-cost-optimization-services-expenses-overview-and-traffic-costs-in-aws/)):

![](https://cdn-images-1.medium.com/max/546/0*-pJVjO4aAl3NC1-M.png)

Wait a few minutes for the Active status:

![](https://cdn-images-1.medium.com/max/1024/0*nxx0RMVhgPL9hxuu.png)

![](https://cdn-images-1.medium.com/max/1024/0*IoipSjvb71Sq1Iru.png)

And connect using AWS CLI by specifying `--connection-type eice`, because the instances are on a private network:

```
$ aws --profile work ec2-instance-connect ssh --instance-id i-011b1c0b5857b0d92 --connection-type eice
...
[ec2-user@ip-10-0-34-239 ~]$
```

### Terraform: EC2 Instance Connect, EKS, and VPC

For the Terraform, here you will need to add the[`node_security_group_additional_rules`](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest#input_node_security_group_additional_rules) in the [EKS module](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/) for SSH access, and create an EC2 Instance Connect Endpoint for the VPC, as in my case we create VPC and EKS separately.

```
...
module "eks" {
  source = "terraform-aws-modules/eks/aws"
  version = "~> 19.21.0"

  cluster_name = local.env_name
  cluster_version = var.eks_version

  ...
      # allow SSM
      iam_role_additional_policies = {
        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }
      ...
  }

  node_security_group_name = "${local.env_name}-node-sg"
  cluster_security_group_name = "${local.env_name}-cluster-sg"

  # to use with EC2 Instance Connect
  node_security_group_additional_rules = {
    ingress_ssh_vpc = {
      description = "SSH from VPC"
      protocol = "tcp"
      from_port = 22
      to_port = 22
      cidr_blocks = [local.vpc_out.vpc_cidr]
      type = "ingress"
    }
  }

  node_security_group_tags = {
    "karpenter.sh/discovery" = local.env_name
  }
  ...
}
...
```

If you created it manually, as described above, then remove the rule from SecurityGroup with SSH and deploy it from Terraform.

For the VPC EC2 Instance Connect Endpoint, I did not find how to do this through Anton Babenko’s module [terraform-aws-modules/vpc](https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest), but you can make it a separate resource through [`aws_ec2_instance_connect_endpoint`](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_instance_connect_endpoint):

```
resource "aws_ec2_instance_connect_endpoint" "example" {
  subnet_id = module.vpc.private_subnets[0]
  security_group_ids = ["sg-0b70cfd6019c635af"]
}
```

However, here you need to pass the SecurityGroup ID from the cluster, and the cluster is created after the VPC, so there is a chicken-and-egg problem.

In general, the Instance Connect seems to be a little more complicated than SSM in the automation, because there are more changes in the code, and in different modules.

However, it is a working option, and if your automation allows it, you can use it.

### Option 3: the old-fashioned way with SSH Public Key via EC2 User Data

And the oldest and perhaps the simplest option is to create an SSH key yourself and add its public part to EC2 when creating an instance.

The disadvantages here are that it will be difficult to add many keys in this way, and EC2 User Data can sometimes go sideways, but if you need to add only one key, a kind of “super-admin” in case of emergency, then this is a perfectly valid option.

Moreover, if you have a VPN to the VPC (see [Pritunl: Launching a VPN in AWS on EC2 with Terraform](https://rtfm.co.ua/en/pritunl-launching-a-vpn-in-aws-on-ec2-with-terraform/)), then the connection will be even easier.

Create a key:

```
$ ssh-keygen 
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/setevoy/.ssh/id_ed25519): /home/setevoy/.ssh/atlas-eks-ec2
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/setevoy/.ssh/atlas-eks-ec2
Your public key has been saved in /home/setevoy/.ssh/atlas-eks-ec2.pub
...
```

The public part can be stored in a repository — copy it:

```
$ cat ~/.ssh/atlas-eks-ec2.pub 
ssh-ed25519 AAA***VMO setevoy@setevoy-wrk-laptop
```

Next, a few crutches: the `EC2NodeClass` in my case is created from the Terraform code through the `kubectl_manifest` resource. The easiest option that has come to mind so far is to add the public key to the `variables`, and then use it in the `kubectl_manifest`.

Later, I will probably move such resources to a dedicated Helm chart.

For now, let’s create a new variable:

```
variable "karpenter_nodeclass_ssh" {
  type = string
  default = "ssh-ed25519 AAA***VMO setevoy@setevoy-wrk-laptop"
  description = "SSH Public key for EC2 created by Karpenter"
}
```

In the `EC2NodeClass` configuration, add the [`spec.userData`](https://karpenter.sh/docs/concepts/nodeclasses/#specuserdata):

```
resource "kubectl_manifest" "karpenter_node_class" {
  yaml_body = <<-YAML
    apiVersion: karpenter.k8s.aws/v1beta1
    kind: EC2NodeClass
    metadata:
      name: default
    spec:
      amiFamily: AL2
      role: ${module.eks.eks_managed_node_groups["${local.env_name_short}-default"].iam_role_name}
      subnetSelectorTerms:
        - tags:
            karpenter.sh/discovery: "atlas-vpc-${var.environment}-private"
      securityGroupSelectorTerms:
        - tags:
            karpenter.sh/discovery: ${local.env_name}
      tags:
        Name: ${local.env_name_short}-karpenter
        environment: ${var.environment}
        created-by: "karpneter"
        karpenter.sh/discovery: ${local.env_name}
      userData: |
        #!/bin/bash
        mkdir -p ~ec2-user/.ssh/
        touch ~ec2-user/.ssh/authorized_keys
        echo "${var.karpenter_nodeclass_ssh}" >> ~ec2-user/.ssh/authorized_keys
        chmod -R go-w ~ec2-user/.ssh/authorized_keys
        chown -R ec2-user ~ec2-user/.ssh       
  YAML

  depends_on = [
    helm_release.karpenter
  ]
}
```

If you are using **not** Amazon Linux, then change the `ec2-user` to the desired one.

**_Important Note_** _: Keep in mind that changes to EC2NodeClass will recreate all instances, and that your services are configured for stable operation, see_ [_Kubernetes: Providing High Availability for Pods_](https://rtfm.co.ua/en/kubernetes-ensuring-high-availability-for-pods/)_._

Deploy it and check it:

```
$ kk get ec2nodeclass -o yaml
...
    userData: #!/bin/bash\nmkdir -p ~ec2-user/.ssh/\ntouch ~ec2-user/.ssh/authorized_keys\necho
      \"ssh-ed25519 AAA***VMO setevoy@setevoy-wrk-laptop\" >> ~ec2-user/.ssh/authorized_keys\nchmod -R go-w
      ~ec2-user/.ssh/authorized_keys\nchown -R ec2-user ~ec2-user/.ssh \n
...
```

Wait for Karpenter to create a new WorkerNode and try SSH:

```
$ ssh -i ~/.ssh/hOS/atlas-eks-ec2 ec2-user@10.0.39.73
...
[ec2-user@ip-10-0-39-73 ~]$
```

Done.

### Conclusions.

- **AWS SessionManager** : looks like the easiest option in terms of automation, recommended by AWS itself, but you need to think about how to use `scp` through it (although it seems to be possible through additional moves, see [.SSH and SCP with AWS SSM](https://globaldatanet.com/tech-blog/ssh-and-scp-with-aws-ssm))
- **AWS EC2 Instance Connect** : a cool feature from Amazon, but somehow more troublesome to automate, so not our option
- **“grandfathered” SSH** : well, the old one is tried and true :-) but I don’t really like User Data, because sometimes it can lead to problems with launching instances; however, it is also simple in terms of automation, and gives you the usual SSH without additional movements

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/aws-karpenter-and-ssh-for-kubernetes-workernodes/)_._

* * *