EKS on AWS re:Invent 2023
Tags: aws, kubernetes, ai, security
Published: 2023-12-03 22:54:38

_Consider supporting my work by [reading this on Medium](https://vyrwu.medium.com/eks-on-aws-re-invent-2023-las-vegas-f46c3440e6af). Thanks!_ ❤

![](https://cdn-images-1.medium.com/max/2400/1*mcDxlY3Ak5VE7FRXrBsJxg.jpeg)

_This is how modern K8s feels like on AWS._

Here’s a collection of my learnings from [**AWS re:Invent 2023 Las Vegas**](https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2023/) — an incredible event for innovators and technologists from all over the world, in a reality bending place which is Las Vegas.

> By 2028, more than 95% of global organisations will be running containerised applications in production, which is a significant increase from fewer than 50% in 2023.

**My focus was** [**AWS EKS — AWS Elastic Kubernetes Service**](https://www.eksworkshop.com/docs/introduction/)**.** I attended a variety of sessions and workshops focused on the latest advancements, emerging tools and solutions for containerised workloads based on [Kubernetes](https://kubernetes.io/):

- Generative AI
- Karpenter
- Security
- Multi-tenancy

**I hope it will become helpful to you!**

---

## Generative AI on EKS

[_Workshop from re:Invent 2023 available for free (thx AWS)._](https://awslabs.github.io/data-on-eks/)

GenAI companies like [OpenAI use Kubernetes](https://openai.com/research/scaling-kubernetes-to-7500-nodes) to train their foundational models like [GPT-3](https://openai.com/blog/gpt-3-apps), [GPT-4](https://openai.com/research/gpt-4), and [DALL·E](https://openai.com/research/dall-e) by distributing model training across thousands of powerful GPU-enabled VMs. **With frameworks like** [**horovod**](https://horovod.readthedocs.io/en/latest/summary_include.html#why-horovod)**, and k8s operators like** [**MPI**](https://github.com/kubeflow/mpi-operator/)**, you can perform distributed deep learning on EKS at scale.** You will need to configure [CUDA time-slicing on nvidia-device-plugin](https://github.com/NVIDIA/k8s-device-plugin#shared-access-to-gpus-with-cuda-time-slicing) to allow GPU sharing across multiple training jobs.

Newly released [Mountpoint for S3 CSI Driver](https://aws.amazon.com/about-aws/whats-new/2023/11/mountpoint-amazon-s3-csi-driver/) adds file system interface access to large training datasets stored on S3 (previously only possible with [FSx for CSI drivers](https://docs.aws.amazon.com/eks/latest/userguide/storage.html)).

Increasingly AWS customers are benefiting from their investments in EKS by baking in complete end-to-end MLOps capabilities into their clusters. [**JARK on EKS**](https://aws.amazon.com/blogs/containers/deploy-generative-ai-models-on-amazon-eks/) **allows you to use** [**Jupiter Notebooks**](https://z2jh.jupyter.org/en/stable/)**,** [**Argo Workflows**](https://argoproj.github.io/argo-workflows/)**, and** [**kuberay**](https://github.com/ray-project/kuberay) **to build, train, test, serve and re-train your AI/ML on EKS.**

![](https://cdn-images-1.medium.com/max/1600/1*x7dhRZTI_C5Dr8Ufl6kQqQ.png)

During AWS re:Invent 2023, by leveraging [EKS Terraform blueprints](https://github.com/aws-ia/terraform-aws-eks-blueprints), in just two hours we deployed an entire EKS cluster with JARK stack installed, and deployed + re-trained [stable diffusion](https://stability.ai/stable-diffusion) (images from prompt) using custom data.

---

## Karpenter

[_Workshop from AWS re:Invent 2023 available for free (thx AWS)._](https://ec2spotworkshops.com/karpenter/010_prerequisites/self_paced.html)

[Karpenter](https://karpenter.sh/), which has [recently graduated to beta](https://aws.amazon.com/blogs/containers/karpenter-graduates-to-beta/), is becoming a de-facto standard for node auto-scaling on EKS.

It’s faster than [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler), because it uses a synchronous [EC2 Fleet API](https://karpenter.sh/docs/faq/#how-does-karpenter-dynamically-select-instance-types) instead of the Auto Scaling Groups. It’s smarter, because it consolidates your nodes down to the most optimal mixture of [node types best fitted to your workloads](https://karpenter.sh/preview/reference/instance-types/)(incl. Spot), rather than simply scaling up the ASG with more nodes of the same type. **This drives down your compute cost while maximising performance and reliability.**

![](https://cdn-images-1.medium.com/max/1600/1*QpYQiMvCb7wiSTJ-FehDWA.jpeg)

It’s important to [follow best practices for resource requests/limits for all EKS Workloads](https://aws.github.io/aws-eks-best-practices/reliability/docs/dataplane/#configure-and-size-resource-requestslimits-for-all-workloads) to avoid OOM Kills due to pod bursts during consolidation (TLDR; memory request = limit).

During the workshop, we simulated load on the cluster and observed Karpenter consolidation using [eks-node-viewer](https://github.com/awslabs/eks-node-viewer), which shows you a real-time breakdown of the EKS nodes, including resource utilisation, number of pods, node type, cost per node, and even an estimated total cost of the cluster per month.

---

## Security

[_Workshop from AWS re:Invent 2023 available for free (thx AWS)._](https://catalog.workshops.aws/detective/en-US/introduction)

[AWS Security Hub](https://docs.aws.amazon.com/securityhub/latest/userguide/what-is-securityhub.html) can ingest findings regarding your Kubernetes resources and Helm chart miss-configurations generated by [kubescape](https://github.com/kubescape/kubescape) and [[AWS Config with Conformance Pack for EKS](https://docs.aws.amazon.com/config/latest/developerguide/security-best-practices-for-EKS.html). Code to translate kubescape controls in S3 to AWS Security Hub findings available [here](https://github.com/vyrwu/kubescape-to-aws-security-hub/blob/main/lambda.py).

![](https://cdn-images-1.medium.com/max/1600/1*DBGxa_uZTZDR45rfid_1BA.jpeg)

[GuardDuty EKS Runtime Monitoring](https://www.eksworkshop.com/docs/security/guardduty/) looks for system-level threats like illegal file access, permission escalations, miner binaries, and suspicious network connections. [GuardDuty EKS Audit Log Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty-eks-audit-log-monitoring.html) generates findings based on the EKS control plane access logs (requires [enabling control plane logs](https://docs.aws.amazon.com/guardduty/latest/ug/features-kubernetes-protection.html#guardduty_k8s-audit-logs)). AWS Inspector scans your container images in ECR for vulnerabilities, and can now also [scan images directly in CI](https://aws.amazon.com/blogs/aws/three-new-capabilities-for-amazon-inspector-broaden-the-realm-of-vulnerability-scanning-for-workloads/), via the new `inspector-scan` API and `inspector-sbomgen` AWS CLI command.

**The kicker here is** [**AWS Detective**](https://docs.aws.amazon.com/eks/latest/userguide/integration-detective.html)**, which looks for patterns across all these findings, plus CloudTrail and VPC flow logs, giving you real-time analysis of threats in your EKS environments, down to individual IPs, IAM Roles and even Kubernetes service accounts and role bindings.**

Check out [Pod Security Standards](https://www.eksworkshop.com/docs/security/pod-security-standards/), and remember that you can enforce them as [kyverno policies](https://kyverno.io/policies/pod-security/).

---

## Multi-tenancy

[_Workshop from AWS re:Invent 2023 available for free (thx AWS)._](https://catalog.us-east-1.prod.workshops.aws/workshops/02d7d511-9af8-417f-abda-3708e9c0c749/en-US)

[SaaS control plane](https://docs.aws.amazon.com/whitepapers/latest/saas-architecture-fundamentals/control-plane-vs.-application-plane.html) sits on top of your SaaS application and manages your tenants, subscriptions, tiering, customer onboarding, third-party integrations, and product observability. It’s a separate layer from the application itself, requiring expertise in access control protocols, infrastructure, telemetry, as well as the API and product development. EKS delivers efficiencies and new monetisation strategies by becoming a central piece in architecting the SaaS control plane.

![](https://cdn-images-1.medium.com/max/1600/1*wWdRPn5L3XBVxCqDgTUw2Q.png)

**Using** [**Taints and Tolerations**](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) **, and** [**Node Affinity**](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) **we can allocate different sets of compute to different tiers.** For example, we can run premium tenants using On-demand EC2s, while free tier users on [Spot](https://aws.amazon.com/ec2/spot/) to save money. [Istio](https://istio.io/) and [Istio Gateway](https://istio.io/latest/docs/reference/config/networking/gateway/) allows you to [route traffic to different application pools based on JWT claims](https://istio.io/latest/docs/tasks/security/authentication/jwt-route/), and using [Karpenter](https://karpenter.sh/) in combination with [HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), we can eliminate noisy-neighbour problems coming from a spiky usage of the free tier while ensuring the desired compute is always available just-in-time, without over-provisioning. We can expose tenant metrics from our SaaS applications to [Prometheus](https://github.com/prometheus/prometheus) and plot them as graphs in [Grafana](https://github.com/grafana/grafana) for unified observability.

---

Hope my collection of notes were helpful to you. **I was fortunate to be a part of AWS re:Invent 2023 thanks to** [**ZeroNorth**](http://zeronorth.com/), where we build modern platforms and capabilities on top of EKS to **make global trade green.** **💚**

![](https://cdn-images-1.medium.com/max/1600/1*bkHn2qqrzbAOjLvWjlb-rQ.jpeg)

_Bloopers: Google Cloud advertising up the Sphere, right next to AWS re:Invent 2023 :D_