Chaos Engineering with LitmusChaos on AWS EKS using IRSA
Tags: aws, chaos, tutorial, cloud
Published: 2022-05-23 14:33:45

<!-- wp:paragraph -->
<p>I have already kicked off a brand-new series on my YouTube channel called <a href="https://youtube.com/playlist?list=PLJ9A48W0kpRKyBBmwOz6oSn4s3A90HHCj" target="_blank" rel="noreferrer noopener">Learn Chaos Engineering series</a>. The first few episodes will be focusing on LitmusChaos. I have been working on LitmusChaos for the past several weeks. The more I work, the more questions arise about the   mechanics of LitmusChaos. In episode 2, we saw an EC2 instance termination experiment using AWS Secrets, which is one way to run chaos on AWS. Exposing secrets and access keys is not a recommended practice. Enter IRSA. IAM Role for Service Account. Instead of creating a video, let me write it as a blog post; I hope it will make a great impression on people who are getting started. Before writing this blog post, I practiced several times. Let us proceed.</p>
<!-- /wp:paragraph -->

<!-- wp:embed {"url":"https://youtube.com/playlist?list=PLJ9A48W0kpRKyBBmwOz6oSn4s3A90HHCj","type":"video","providerNameSlug":"youtube","responsive":true,"className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://youtube.com/playlist?list=PLJ9A48W0kpRKyBBmwOz6oSn4s3A90HHCj
</div></figure>
<!-- /wp:embed -->

<!-- wp:generateblocks/headline {"uniqueId":"f6d73ebc"} -->
<h2 class="gb-headline gb-headline-f6d73ebc gb-headline-text">Prerequisites</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>There are several moving components when it comes to AWS EKS. Few are optional, but it would be a great add-on if you hit a roadblock. e.g. I am using <code>Octant</code> to visualize the Kubernetes ecosystem. You can use Lens or Kubernetes Dashboard if you are familiar with them.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Below are the stuffs you need before you proceed:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>eksctl to spin up AWS EKS cluster</li><li>LitmusChaos</li><li>AWS IAM Role and Policy creation access</li><li>Helm (optional)</li><li>Octant (optional)</li><li>EC2 instance to run the experiments on it.</li></ol>
<!-- /wp:list -->

<!-- wp:generateblocks/headline {"uniqueId":"0a9163c6"} -->
<h2 class="gb-headline gb-headline-0a9163c6 gb-headline-text">What is LitmusChaos?</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Litmus is an open source Chaos Engineering platform that enables teams to identify weaknesses &amp; potential outages in infrastructures by inducing chaos tests in a controlled way.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Recently, it was acquired by Harness. It is an CNCF Incubation project.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"b8044320"} -->
<h2 class="gb-headline gb-headline-b8044320 gb-headline-text">eksctl</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p><code>eksctl</code> is a great CLI utility to perform various AWS EKS tasks such as cluster management, hardening the security, GitOps and more. It is written in Go and uses CloudFormation under the hood. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Using <code>eksctl</code> we can create a cluster in one command. But I do not recommend that. e.g. it will create two <code>m5.large</code> worker nodes on <code>us-west-2</code> which we do not want for learning purpose. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Instead, use the <code>--dry-run</code> command to output to a file and modify the parameters such as region, instance type, availability zone etc.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Install <code>eksctl</code> from <a href="https://eksctl.io/introduction/#installation" target="_blank" rel="noreferrer noopener nofollow">this documentation</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter the below command which will create a EKS yaml manifest.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>eksctl create cluster --name=chaos-playground --region=us-east-1 --dry-run &gt; chaos-playground.yaml</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The name of our cluster is <code>chaos-playground</code> and will be created on <code>us-east-1</code>. Change the region based on your location so that you will reach the nearest availability zone. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Open the yaml manifest in your favorite editor e.g. <code>vim chaos-playground.yaml</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Change the instance type to <code>t3.medium</code>. Save it and apply the below command to start creating a cluster.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>eksctl create cluster -f chaos-playground.yaml</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Go for a coffee or tea or your favorite beverage and come back after 15-25 minutes. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Once the EKS is up and running, you will get the below message.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>EKS cluster "chaos-playground" in "us-east-1" region is ready</code></p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"eb825e38","element":"h3"} -->
<h3 class="gb-headline gb-headline-eb825e38 gb-headline-text">Updating kube config</h3>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>You might be running various clusters locally or in the cloud. Let us update the kube config current context using the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>aws eks update-kubeconfig --name chaos-playground --region us-east-1</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The above command change the context to <code>chaos-playground</code>. To validate the current context, enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k config current-context</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I have an alias <code>k</code> for <code>kubectl</code>. You may need to enter <code>kubectl</code> in place of <code>k</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Install <code>kubectl</code> from <a href="https://kubernetes.io/docs/tasks/tools/" target="_blank" rel="noreferrer noopener nofollow">here</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"8a70133b"} -->
<h2 class="gb-headline gb-headline-8a70133b gb-headline-text">LitmusChaos</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Let us begin by installing LitmusChaos in <code>litmus</code> namespace. You can install LitmusChaos from <a href="https://docs.litmuschaos.io/docs/getting-started/installation" target="_blank" rel="noreferrer noopener nofollow">here</a>. By default, the services <code>litmusportal-frontend-service</code> and <code>litmusportal-server-service</code> are exposed as <code>NodePort</code>. We need to expose it as <code>ClusterIP</code>. I have already changed the type and kept it in my <a href="https://github.com/QAInsights/Learn-Chaos-Engineering-Series/blob/main/LitmusChaos-AWS-EKS/litmuschaos-2.9.0.yaml" target="_blank" rel="noreferrer noopener">GitHub repo</a>. Let us apply that to our cluster using the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k apply -f https://raw.githubusercontent.com/QAInsights/Learn-Chaos-Engineering-Series/main/LitmusChaos-AWS-EKS/litmuschaos-2.9.0.yaml</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The above command will create various Kubernetes objects as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9390,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-10.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-10-1024x440.png" alt="" class="wp-image-9390"/></a><figcaption>Installing LitmusChaos</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>By default, all the Litmus Custom Resource Definitions (CRDs) will be installed in the <code>litmus</code> namespace.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let us verify the pods which are running in the <code>litmus</code> namespace.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k get po -n litmus</code></p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":9391,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-11.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-11.png" alt="k get po -n litmus" class="wp-image-9391"/></a><figcaption><code>k get po -n litmus</code></figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The next step is to expose the service <code>litmusportal-frontend-service</code> to get the load balanced URL.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter the below command to patch the service.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k patch svc litmusportal-frontend-service -p '{"spec": {"type": "LoadBalancer"}}' -n litmus</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To get the load balanced URL, enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k get -n litmus svc litmusportal-frontend-service -o wide</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You will get the URL and port. e.g. you can access the LitmusChaos UI using port 9091 and URL <code>http://a336e7fc0d2d64029abdf53d95aa1cca-97284119.us-east-1.elb.amazonaws.com:9091</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let us wait for a few moments before the URL is reachable. Meanwhile, let us install Octant from <a href="https://github.com/vmware-tanzu/octant/releases" target="_blank" rel="noreferrer noopener nofollow">here</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Run the Octant program and launch <code>127.0.0.1:7777</code> the homepage of Octant. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The first step is to configure the kube config. Go to <strong>Preferences</strong> <strong>&gt; General</strong> and note down the kube config path. Make sure our AWS EKS kube config is present in the Octant kube config. If it is not present, Octant will not display the <code>chaos-playground</code> objects.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here are the pods which are running in <code>litmus</code> namespace on <code>chaos-playground</code> cluster.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9392,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-12.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-12-1024x305.png" alt="Litmus Pods" class="wp-image-9392"/></a><figcaption>Litmus Pods</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Now it is time to launch the LitmusChaos UI. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The default credential is <code>admin/litmus</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Upon the first time login, LitmusChaos will prompt you to change the password. You can skip it or change the credentials.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After successful login, click <code>ChaosAgents</code> to validate the <code>Self-Agent</code> status.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Wait till the <code>Self-Agent</code> is in <code>Active</code> status as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9393,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-13.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-13.png" alt="" class="wp-image-9393"/></a><figcaption>ChaosAgent</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Congratulations! LitmusChaos is up and running now. Let us run an experiment. Before we begin running the experiments, we need to form a hypothesis.</p>
<!-- /wp:paragraph -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>A single EC2 instance running with no auto-scaling group.</p><cite>Provided</cite></blockquote>
<!-- /wp:quote -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>Failure of a single EC2 instance will disrupt the customer experience.</p><cite>Hypothesis</cite></blockquote>
<!-- /wp:quote -->

<!-- wp:generateblocks/headline {"uniqueId":"e5441455"} -->
<h2 class="gb-headline gb-headline-e5441455 gb-headline-text">Spin an EC2 Instance</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>If you are new to AWS, <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html" target="_blank" rel="noreferrer noopener nofollow">please follow this tutorial</a> to spin an EC2 instance in your region.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For this demo, I have spun up an EC2 instance with t2.micro which is running a nginx server.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9395,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-14.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-14.png" alt="EC2 Instance up and running" class="wp-image-9395"/></a><figcaption>EC2 Instance up and running</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:generateblocks/headline {"uniqueId":"53fece83"} -->
<h2 class="gb-headline gb-headline-53fece83 gb-headline-text">EC2 Terminate By ID Experiment</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Let us go back to LitmusChaos and create a new experiment which will terminate the above EC2 instance using its ID.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Litmus Workflows &gt; Schedule a workflow </strong>to start creating a workflow. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Select the <strong>Self-Agent</strong>, then click <strong>Next</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Create a new workflow using the experiments from ChaosHubs</strong> radio button and select <strong>Litmus ChaosHub</strong>, then click <strong>Next</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Next </strong>again, leaving the default values for the workflow name.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Add a new experiment</strong> and search for <code>ec2</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Select <code>kube-aws/ec2-terminate-by-id</code> and then click <strong>Done</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9397,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-15.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-15.png" alt="Select experiment" class="wp-image-9397"/></a><figcaption>Select experiment</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Click the pen icon to edit the experiment.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9398,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-16.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-16-1024x433.png" alt="Edit YAML" class="wp-image-9398"/></a><figcaption>Edit YAML</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Click <strong>Next </strong>three times, leaving the default options.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Under the <strong>Tune Experiment </strong>section, enter the <strong>EC2 Instance ID.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click <strong>Show more environment variables</strong> button to enter the region, then click <strong>Finish</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9399,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-17.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-17.png" alt="Tune Experiment" class="wp-image-9399"/></a><figcaption>Tune Experiment</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Click <strong>Next </strong>few times and then <strong>Finish </strong>to start the experiment. This experiment will fail after a while.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9400,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-18.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-18-1024x293.png" alt="" class="wp-image-9400"/></a><figcaption>Failed Experiment</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Why it got failed?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>LitmusChaos objects doesn't have any access to kill the EC2 instance. Even though we are running everything on the same AWS account. To authenticate LitmusChaos objects to the AWS resources, there are two ways: using AWS Secrets or using IAM service.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Mounting AWS secrets is not a recommended practice for running the experiment. As we are exposing the secrets in YAML file. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter IRSA. IAM Role for Service Account.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>IAM is a web service that helps you securely control access to AWS resources. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Service Account in this context is meant for Kubernetes. A Kubernetes service account provides an identity for processes that run in a pod.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The following are the steps which are involved in creating IAM roles for service account.</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Create OIDC provider</li><li>Create IAM Role and Policy</li><li>Associate IAM Role</li></ol>
<!-- /wp:list -->

<!-- wp:generateblocks/headline {"uniqueId":"efeaddd7","element":"h3"} -->
<h3 class="gb-headline gb-headline-efeaddd7 gb-headline-text">Create OIDC provider</h3>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>The OIDC concept is beyond the scope of this article. On a high level, OpenID Connect authenticates the AWS API with the supported identity providers and receives a JWT token. This token can be passed into AWS STS to receive temp credentials to authenticate into any AWS service like EC2.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter the below command to validate the OIDC provider for the cluster.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>aws eks describe-cluster --name chaos-playground --region us-east-1 --query "cluster.identity.oidc.issuer" --output text</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This will list the following output.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>https://oidc.eks.us-east-1.amazonaws.com/id/B718311B05C5C27CCF96C406CEXXXXXX</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If there is no output, enter the below command to create one.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code><strong>eksctl utils associate-iam-oidc-provider --cluster </strong><code>chaos-playground</code><strong> --region us-east-1 --approve</strong></code></p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"4c94405c"} -->
<h2 class="gb-headline gb-headline-4c94405c gb-headline-text">Create IAM Policy and Role</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Copy the below policy and save it as <code>chaos-playground.json</code>. Replace the account ID with your account ID.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": &#91;
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:*",
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sts:*",
            "Resource": "arn:aws:iam::&lt;your-account-id&gt;:role/*"
        }
    ]
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To create a policy, enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>aws iam create-policy --policy-name ChaosPlaygroundPolicy --policy-document file://chaos-playground-policy.json</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Each entity in AWS has a unique identifier called ARN. To retrieve the ARN for the above policy, enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>aws iam list-policies --query 'Policies[?PolicyName==ChaosPlaygroundPolicy].Arn' --output text</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Copy the output which we are going to need it in the subsequent step.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The next step is to create a Trust Policy for our IAM role. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Save the below trust policy as <code>chaos-playground-trust.json</code> and make sure you replace the OIDC value appropriately.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>{
	"Version": "2012-10-17",
	"Statement": &#91;
		{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::&lt;your-account-id>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/B718311B05C5C27CCF96C406XXXXXXX"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringEquals": {					
					"oidc.eks.us-east-1.amazonaws.com/id/B718311B05C5C27CCF96C406XXXXXXX:aud": "sts.amazonaws.com"
				}
			}
		}
	]
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><code>aws iam create-role --role-name Chaos-Playground-Role --assume-role-policy-document file://chaos-playground-trust.json</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Once the role is created with the trust policy, the next step is to attach a policy which we created earlier, i.e. <code>ChaosPlaygroundPolicy</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To attach it to the IAM role, enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>aws iam attach-role-policy --policy-arn arn:aws:iam::</code>&lt;your-account-id&gt;<code>:policy/ChaosPlaygroundPolicy --role-name Chaos-Playground-Role</code></p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"c18e430d"} -->
<h2 class="gb-headline gb-headline-c18e430d gb-headline-text">Associate IAM Role</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>To associate the IAM Role to the Kubernetes Service Account, the first step is to create a service account using eksctl command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter the below command to create a IAM SA.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>eksctl create iamserviceaccount --cluster=chaos-playground --namespace=litmus --name=ec2-terminate-sa-litmus --attach-policy-arn="arn:aws:iam::&lt;your-account-d&gt;:policy/ChaosPlaygroundPolicy" --override-existing-serviceaccounts --approve --region us-east-1</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To validate all the service accounts in <code>litmus</code> namespace. Enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k get sa -n litmus</code></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9404,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-20.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-20.png" alt="k get sa -n litmus" class="wp-image-9404"/></a><figcaption><code>k get sa -n litmus</code></figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>After creating a service account, the next step is to annotate the service account with the IAM role ARN using the below command. The below command will annotate the <code>litmus-admin</code> service account. This will suffice for our experiment.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k annotate serviceaccount -n litmus litmus-admin eks.amazonaws.com/role-arn=arn:aws:iam::202835464218:role/Chaos-Playground-Role --overwrite</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To validate the annotations, enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>k describe sa -n litmus ec2-terminate-sa-litmus</code></p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":9409,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-23.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-23-1024x192.png" alt="" class="wp-image-9409"/></a></figure>
<!-- /wp:image -->

<!-- wp:generateblocks/headline {"uniqueId":"95aba71c"} -->
<h2 class="gb-headline gb-headline-95aba71c gb-headline-text">LitmusChaos Experiments</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Let us run the experiment again. But before triggering it, we need to remove the mounted secrets from our experiment. As we are using IRSA, we do not need to mount it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Head to <strong>LitmusChaos &gt; Litmus Workflows &gt; Schedules</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click the vertical three dots and then click <strong>Download Manifest</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9414,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-26.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-26.png" alt="" class="wp-image-9414"/></a><figcaption>Download Manifest</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Open the manifest in your favorite editor. Remove the lines which has <code>secrets</code>. Save it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Create a new experiment by upload the YAML as shown below. Then, start the execution.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9415,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-27.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-27-1024x758.png" alt="" class="wp-image-9415"/></a><figcaption>Rerun the experiment</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>If all is good, this time, our experiment will pass.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>While the experiment is running, you can check the logs from Octant.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9416,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-28.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-28-1024x557.png" alt="" class="wp-image-9416"/></a><figcaption>Octant Logs</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Or check the status in EC2 Instances dashboard.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9417,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-29.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-29.png" alt="EC2 Stopped" class="wp-image-9417"/></a><figcaption>EC2 Stopped</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Once the experiment is completed, LitmusChaos will revert the state of the EC2 instance.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9418,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-30.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-30.png" alt="Reverted Status" class="wp-image-9418"/></a><figcaption>Reverted Status</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here is the LitmusChaos workflow graph view.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9419,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-31.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-31-1024x332.png" alt="" class="wp-image-9419"/></a><figcaption>Experiment Pass</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Thanks for staying with me :) </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Important Notes</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>The policy we created basically opens for all the resources and the action, you must fine tune it for better security. Since it is a demo, I was being playful.</li><li>The annotations are for all the service accounts. I yet to validate which SA needs to be annotated properly.</li><li>Once the experiment is done, make sure you terminate the cluster and other resources.</li></ul>
<!-- /wp:list -->

<!-- wp:generateblocks/headline {"uniqueId":"f3067676"} -->
<h2 class="gb-headline gb-headline-f3067676 gb-headline-text">Conclusion</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>IRSA is a beautiful implementation of a zero-trust framework. LitmusChaos is architected to answer the needs of enterprise security who are in the AWS and GCP ecosystem. We have just scratched the surface of LitmusChaos. Eventually, I will cover the other experiments in my channel.</p>
<!-- /wp:paragraph -->