How to reduce your Amazon EKS costs by half in 15 minutes
Tags: devops, aws, kubernetes, eks
Published: 2021-06-28 11:17:24

<!-- wp:tadv/classic-paragraph -->
<p>Overprovisioning is the top reason why teams see their cloud bills constantly growing. But choosing the best instances from the hundreds of options AWS offers is a tough call.</p>
<p>How are you supposed to know which ones will deliver the performance you need?</p>
<p>Fortunately, you can find solutions on the market that can do that for you.</p>
<p>If you’re curious about how it all works, follow my journey and see how I slashed the costs of running my containers in EKS by half using CAST AI, in 15 minutes.</p>
<p><br><strong>TL;DR</strong></p>
<p>I started by provisioning an e-commerce app <a href="https://github.com/castai/examples">(here)</a> on an EKS cluster with six m5 nodes (2 vCPU, 8 GiB) on AWS EKS. I then deployed an AI engine to analyze my application and suggest some optimizations. Finally, I activated the engine and watched the system self-optimize.</p>
<p>The initial cluster cost was <span style="color: #ff0000;"><strong>$414</strong></span> per month. Within 15 minutes, in a fully automated way, the cluster cost went to <span style="color: #008000;"><strong>$207</strong></span> (a 50% reduction), by reducing six nodes to three nodes. Then, 5 minutes later, the cluster cost went down to <span style="color: #008000;"><strong>$138</strong></span> per month, using spot instances (a 66% reduction).</p>
<!-- /wp:tadv/classic-paragraph -->

<!-- wp:image {"id":2417,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://cast.ai/wp-content/uploads/2021/06/temp-slide-doc-1-1200x376.png" alt="" class="wp-image-2417"/></figure>
<!-- /wp:image -->

<!-- wp:tadv/classic-paragraph -->
<p>&nbsp;</p>
<p><span style="font-weight: 400;">Get your free </span><a href="https://cast.ai/eks-optimizer/"><span style="font-weight: 400;">CAST AI Savings Report</span></a><span style="font-weight: 400;"> to check how much you could potentially save. It’s the best starting point for any journey into cloud cost optimization.<br></span></p>
<h2><span style="font-weight: 400;">Step 1: Deploying my app and running the Savings Report</span></h2>
<p><span style="font-weight: 400;">&nbsp;</span><span style="font-size: inherit; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;">I deployed my app in 6 nodes on EKS. Here’s what it looked like before - all the nodes were empty:<br></span></p>
<!-- /wp:tadv/classic-paragraph -->

<!-- wp:image {"id":2418,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://cast.ai/wp-content/uploads/2021/06/2-1200x269.png" alt="" class="wp-image-2418"/></figure>
<!-- /wp:image -->

<!-- wp:tadv/classic-paragraph -->
<p class="c5"><span class="c6">The cluster was created via eksctl:<br /></span></p>
<table style="border-collapse: collapse; width: 100%; height: 43px;">
<tbody>
<tr style="height: 43px;">
<td style="width: 100%; height: 43px;"><code><span class="c10">eksctl create cluster --name boutique-blog-lg -N 6 --instance-types m5.large --managed --region us-east-2</span></code></td>
</tr>
</tbody>
</table>
<p class="c5"><span class="c6">And after deployment - the green rectangles are the pods:</span></p>
<p class="c5"><img title="" src="https://lh3.googleusercontent.com/qb9paP7pJE40iBtTHRyZVwF9gQJJ3vTezh-ieuj6Hd4E9QRdnm900-8PhDnxxmx5lCIrO6R20TlYLO3o_aSKzOnycQArU-wSJGR9g8VZyeKt3gbXqlndwF9Agl6aF-TOODSDcdsh" alt="" /></p>
<p class="c5">I’m using <span class="c9"><a class="c4" href="https://www.google.com/url?q=https://github.com/hjacobs/kube-ops-view&amp;sa=D&amp;source=editors&amp;ust=1624475503193000&amp;usg=AOvVaw0oWsk7z3xUO5bg0HNblkO0">kube-ops-view</a></span><span class="c6">, a useful open-source project to visualize the pods.</span></p>
<p class="c5"><span class="c6">Notice that with Kubernetes, the application’s pods (aka containers) are spread evenly across all the nodes by default. Kubernetes is a fair orchestration engine. The CPUs range between 40 and 50%.</span></p>
<p class="c5"><span class="c6">Note: all EKS autoscalings have been disabled on purpose, as CAST AI will substitute EKS Cluster Autoscaling.</span></p>
<p class="c5">Now it’s time to connect my EKS cluster to CAST AI. I created a free account on <a href="https://cast.ai">https://<span class="c18">cast.ai </span></a><span class="c6">and selected the Connect your cluster option.</span></p>
<p class="c5"><strong><span class="c6">Click on ‘Connect cluster’:</span></strong></p>
<p class="c5"><img title="" src="https://lh3.googleusercontent.com/MbbS6UiGKDvgqv3eyxOS9QzTVY301zYGju9MZkSINHhN47eksKPo_eLL0_pQqmBCifauRCwoJULkukKOnY98g7bC8Mq5Rp9QRaB0kaGGkEsM8pD0aBBAMxS5jWW5_bGlePDhjOgq" alt="" /></p>
<p class="c5"><img class="alignnone size-full wp-image-2430" src="https://cast.ai/wp-content/uploads/2021/06/image-1-1.png" alt="" width="1200" height="736" /></p>
<p class="c5">I copied and ran the script successfully on my terminal (I use <span class="c9"><a class="c4" href="https://www.google.com/url?q=https://k8slens.dev/&amp;sa=D&amp;source=editors&amp;ust=1624475503196000&amp;usg=AOvVaw0U3BaFj_ngW1kT6D1Kpyop">Lens</a></span><span class="c6">, another free tool for this).</span></p>
<table style="border-collapse: collapse; width: 100%;">
<tbody>
<tr>
<td style="width: 100%;"><code>laurent@laurents-MacBook-Pro ~ % curl -H "Authorization: Token  "https://api.cast.ai/v1/agent.yaml?provider=eks" | kubectl apply -f -<br />% Total % Received % Xferd Average Speed Time Time Time Current<br />Dload Upload Total Spent Left Speed<br />100 2585 0 2585 0 0 2391 0 --:--:-- 0:00:01 --:--:-- 2391<br />namespace/castai-agent created<br />serviceaccount/castai-agent created<br />clusterrole.rbac.authorization.k8s.io/castai-agent created<br />clusterrolebinding.rbac.authorization.k8s.io/castai-agent created<br />secret/castai-agent created<br />deployment.apps/castai-agent created<br /></code></td>
</tr>
</tbody>
</table>
<p class="c5"><span class="c6"> </span><span class="c6">The CAST AI agent went over my EKS cluster in read-only mode and generated this Savings Report:</span></p>
<p class="c5"><img title="" src="https://lh6.googleusercontent.com/SbkaJvWnIGZgBydgr8hEV4MQ7oLQOivukKUpz27s00KqQxdw0_26z92Bqfo36GY7Enz2VwoKvbIbRavmhnydZ7f8inQrj7oxxC9qWGOYDSMHWHiRj44a3lsqmmh06y15362HvxaQ" alt="" /></p>
<p class="c5"><span class="c6">If I switch my six m5.large to what CAST AI recommends - three c5a.large - I could slash my bill by almost 60%. Sounds like a plan!</span></p>
<p class="c5"><span class="c6">With Spot Instances, I can get even higher savings (66.5%).</span></p>
<p class="c5"><img title="" src="https://lh5.googleusercontent.com/9GN5PEc3L4NAV8KJH354TmTVj7idthFOVFiyqKb-AwvV1rh0T-HQa3zFrjZO8JTZSb8Vn-D4y4d5CWjzWZVmpu40OdPtqgRSWApNhT1WVEqWnnS1mIAuBotAp2u8uUd7vxsBzIS3" alt="" /></p>
<h2 id="h.un2kxzcqtv1n" class="c24"><span class="c7">Step 2: Activating the cost optimization</span></h2>
<p class="c5"><span class="c6">To get started with cost optimization, I need to add my AWS access key ID and Secret access key to the platform.</span></p>
<p class="c5"><span class="c6"> </span><span class="c6">Getting the access keys was easy. All it took was running this script:</span></p>
<p class="c5"><img title="" src="https://lh4.googleusercontent.com/ccEpmkKCYU86QXMz1f_JgNTTfq272LJg-bcn3MsLG2l7500v-yyEgjz942a7BUaiPgq8vD8Z3m1BR9Cmd75bsQvC2Lq9XqhFPsHVM7-zUCEWGPH_600aTnSyad9FoQfxPx5Wn6Is" alt="" /></p>
<h2 id="h.bligzf8rp29x" class="c24">Step 3:<span class="c7"> Enabling policies</span></h2>
<p class="c15"><strong><span class="c6">I turn on all the policies available in CAST AI:</span></strong><img title="" src="https://lh4.googleusercontent.com/lNWeK3KhwBuG4hHv3M7E0i7nCueZOP8qPDY-Bk2J8niwI37BpKUtMQVeW6Gh2psthCbrw_JXh5bs5W1yN_CebiB7d6wjDXqY0de6_RAVJQemiTDThuKH9muKBC-LsaOa--y9frRA" alt="" /></p>
<ul class="c8 lst-kix_i8wh737upq-0 start">
<li class="c15 c20 li-bullet-0"><span class="c13">CPU Policy</span><span class="c6">: I tell the engine to never go above my budget. I set it up to 200 CPUs.</span></li>
<li class="c15 c20 li-bullet-0"><span class="c13">Node autoscaler</span><span class="c6">: CAST AI will make a smart selection of nodes whenever I have unscheduled pods, starting with Spot Instances (if the unscheduled pods are Spot Instance-friendly) or On-Demand.</span></li>
<li class="c15 c20 li-bullet-0"><span class="c13">Node Deletion</span> + <span class="c13">Evictor</span><span class="c6">: Evictor is a background process that continuously shrinks the cluster to the minimum number of nodes by bin-packing pods. Once a node becomes empty, it’s deleted automatically.</span></li>
</ul>
<p class="c5"><span class="c9"><a class="c4" href="https://www.google.com/url?q=https://docs.cast.ai/guides/start-saving-quickly/%23install-evictor-continues-improvements&amp;sa=D&amp;source=editors&amp;ust=1624475503206000&amp;usg=AOvVaw0luzJIPo_a0faJv0OPoYor">Evictor</a></span><span class="c6"> is a CAST AI tool that constantly looks for inefficiencies in a setup and helps to maximize savings as fast as possible. Run this command to activate Evictor:</span></p>
<table style="border-collapse: collapse; width: 100%;">
<tbody>
<tr>
<td style="width: 100%;">
<p><span style="font-size: inherit; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;"> So, I installed Evictor and set it to work.<span class="c16" style="font-size: inherit; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;"> </span><strong style="font-size: inherit; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;"><span class="c6">Evictor is running!</span></strong></span></p>
<p><img style="font-size: inherit; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;" title="" src="https://lh6.googleusercontent.com/O9owmGWLor8DdDiS0qL5IOn_wS2DbmmJn9Piuckt3BGamBJ5HeiNtDNnfefk-bOLVkTAMpuzlaBZKt9U5wuqZ4HYLl90k8o4zVxuEdIjE6AkcoBiv2NIUCNESx2ok1lXJt6JKokO" alt="" /></p>
</td>
</tr>
</tbody>
</table>
<p class="c5"><span class="c6"><br />This is what Evictor in action looks like:</span></p>
<ol class="c8 lst-kix_hi3yi4kp62i4-0 start" start="1">
<li class="c5 c20 li-bullet-0">One node (in <span class="c21">red</span><span class="c6"> below) is identified as a candidate for eviction.</span></li>
<li class="c5 c20 li-bullet-0"><span class="c6">Evictor automatically moves the pods to other nodes “bin-packing.”</span></li>
<li class="c5 c20 li-bullet-0"><span class="c6">Once the node becomes empty, it’s deleted from the cluster.</span></li>
<li class="c5 c20 li-bullet-0"><span class="c6">Go back to step 1.<br /></span></li>
</ol>
<p class="c5"><img title="" src="https://lh5.googleusercontent.com/soZO80e7TalBovKFSPOSgwFm7SzuxyKyfXlFTfl6AQesQsHAEe61SoEShuIg4sdAY4JvzrTZMcVAvVexqaEXnigO2sGyTInN1qWgVCi9uhtGsRZfqrh3UQBDA-orqJeh-PTFUKX0" alt="" /></p>
<p class="c5"><span class="c6">One node is deleted:</span></p>
<p class="c5"><img title="" src="https://lh5.googleusercontent.com/9bpUGSw8s9mK7yQJGoLODNmXxfz85cmxRjcVlElHiJ-_mKnckWJR7yb-nxll4HC8qmb2TBpCIptgSlLkztD0CuXVqc1vVT9zVw4w08RVdLZbJ0CZwAJnx3s5MHSUiad5-TS1eiC-" alt="" /></p>
<p class="c5"><span class="c6">Here are the Evictor logs:</span></p>
<table style="border-collapse: collapse; width: 100%;">
<tbody>
<tr>
<td style="width: 100%;"><code>time="2021-06-14T16:08:27Z" level=debug msg="will try to evict node \"ip-192-168-66-41.us-east-2.compute.internal\""<br />time="2021-06-14T16:08:27Z" level=debug msg="annotating (marking) node \"ip-192-168-66-41.us-east-2.compute.internal\" with \"evictor.cast.ai/evicting\"" node_name=ip-192-168-66-41.us-east-2.compute.internal<br />time="2021-06-14T16:08:27Z" level=debug msg="tainting node \"ip-192-168-66-41.us-east-2.compute.internal\" for eviction" node_name=ip-192-168-66-41.us-east-2.compute.internal<br />time="2021-06-14T16:08:27Z" level=debug msg="started evicting pods from a node" node_name=ip-192-168-66-41.us-east-2.compute.internal<br />time="2021-06-14T16:08:27Z" level=info msg="evicting 9 pods from node \"ip-192-168-66-41.us-east-2.compute.internal\"" node_name=ip-192-168-66-41.us-east-2.compute.internal<br />I0614 16:08:28.831083 1 request.go:655] Throttling request took 1.120968056s, request: GET:https://10.100.0.1:443/api/v1/namespaces/default/pods/shippingservice-7cd7c964-dl54q<br />time="2021-06-14T16:08:44Z" level=debug msg="finished node eviction" node_name=ip-192-168-66-41.us-east-2.compute.interna<br /></code></td>
</tr>
</tbody>
</table>
<p class="c5"><span class="c6">And now the second and third nodes were evicted - 3 nodes remain:</span></p>
<p class="c5"><img title="" src="https://lh5.googleusercontent.com/e5ave8JBWcUVjWuQQ4SsbFy3Bpt7LsvbvOg2ATuIxxigJgPprbjfUMvoOFXEcltw87FV-jsVI-rev3eibHLvlojqjuQ3WFluMjQp7tAkQmQZ78jEHPCuCYw2iTgmcSsj2pjmEMss" alt="" /></p>
<p class="c5"><span class="c6">After about 10 minutes, Evictor deleted 3 nodes and left 3 nodes running. Note that CPUs are now at a much healthier 80% rate.</span></p>
<p class="c5"><span class="c6">The cost of this cluster is now $207.36 per month - half the initial cost of $414 per month.</span></p>
<p class="c5">I managed to achieve <span class="c18 c31">80% of the projected savings</span><span class="c6">. This is what I saw in my CAST AI dashboard:</span><span class="c6"> </span></p>
<p class="c5"><img title="" src="https://lh6.googleusercontent.com/oyKftAWYah-q3KQx7iSSkptiIM90HZUg1wQahZO1uD1bt4HN_2CC34-_rS9TNRLY4YDkG2ZYJpe9MkNXO4TwNR1Q-I-8Ugr7A-uFaX0BRTfH5AR6Hh-ymRT3X_PoGtU36UYKjhD6" alt="" /></p>
<h1 id="h.fxjgvp9hjno7" class="c11"><span class="c23">Advanced savings</span></h1>
<h2 id="h.isx5k3pp5pr3" class="c24"><span class="c7">Step 4 (optional): Moving my app to new optimized nodes</span></h2>
<p class="c5"><span class="c6">Steps 1, 2, and 3 are fully automated. CAST AI gradually shrinks the cluster by eliminating waste and overprovisioning. It does so by bin-packing pods and emptying nodes one by one. From that moment, the cluster is optimized and Evictor will continuously look for further optimization opportunities over time.</span></p>
<p class="c5"><span class="c6">Step 4 is an advanced optional step where CAST AI actively replaces the current nodes with more optimized nodes, such as Spot Instances. The concept is fairly simple: CAST AI cordons the cluster, drains nodes, and replaces them with more optimized nodes.</span></p>
<p class="c5"><span class="c6">The nodes are cordoned:</span></p>
<p class="c5"><img title="" src="https://lh4.googleusercontent.com/61sDEBrhPKMgLjPPHdbnASItTPjK20anWxx9Aj8zKpAB53oUZQxKrcIUE4ZF5Ml2cSW93HBrCS6xbul5Wa8Gpou09M4qt8XsAldU092djrbWPrVUyqmvdFY-uewr1iaOggdn0HNX" alt="" /></p>
<p class="c5"><span class="c6">The first two nodes are drained, and the AI engine selects the most appropriate instances type for these nodes. This is what I saw in my CAST AI dashboard:</span></p>
<p class="c5"><img title="" src="https://lh4.googleusercontent.com/TNdseUf8qK0k1oYASx3IGjH6ECrYZvC6rZ-M9DgH0howjX5LsEOM88_hXytU-ORAP_u2gQ1nEhX-ipSMRGRmC_CzlTzCMQ-rcVTxGvW5mlyOi9dM1T22WV-t46npYhZY_ImiGq9H" alt="" /></p>
<p class="c5">As you can see, my cluster now has only two nodes and costs <span class="c13">$138 per month</span><span class="c6">. It’s hard to imagine that I started out with a monthly EKS bill of $414.72!</span></p>
<h2 id="h.ppsw8eszptox" class="c24"><span class="c7">Summary</span></h2>
<p class="c5">Moving from a non-optimized setup to a fully-optimized one was a breeze. CAST AI analyzed my setup, found opportunities for savings, and swiftly optimized my cluster in 15 minutes. <strong>I cut my EKS bill by half in 15 minutes, from $414 to <span class="c13">$207</span><span class="c6">.</span></strong></p>
<p class="c5">Then, I activated advanced savings by asking CAST AI to replace nodes with more optimized nodes and achieved further savings, ending up with a <span class="c13">$138 </span>bill<span class="c6">.</span></p>
<p class="c5"><img title="" src="https://lh5.googleusercontent.com/EF3vaiUipxuh9kzaNIkJSgmbm7XZhlD4JpjkYloE9g0YcwMXRX4Ug1vzdbur_TeTV5W3xvk1BBSzUJbP0MktTtEzvwgwPVaVffhPyfjbW2vQPSiJly-5G7UtXX1vqyMsVQUsl2xv" alt="" /></p>
<p class="c5">Run the free <span class="c9"><a class="c4" href="https://www.google.com/url?q=https://cast.ai/eks-optimizer/&amp;sa=D&amp;source=editors&amp;ust=1624475503221000&amp;usg=AOvVaw2Joj5x6eWu6VUFs42bNHR2">CAST AI Savings Report</a></span><span class="c6"> to check how much you could potentially save. It’s the best starting point for any journey into cloud cost optimization.</span></p>
<!-- /wp:tadv/classic-paragraph -->