Chaos Engineering with LitmusChaos on AWS - Cross Account Implementation
Tags: aws, cloud, tutorial, kubernetes
Published: 2022-05-27 15:16:53

<!-- wp:paragraph -->
<p>In our last post, we learned about LitmusChaos, how to get started with a simple experiment, IAM Role for Service Account and more. In this blog post, we are going to see how you can run chaos experiments using LitmusChaos on AWS, if you have two different AWS accounts.  Let us get started.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The deployment to production happens at a fast pace in a carefully constructed value stream. Primarily, we (performance engineers) work in a non-production environment. Though we run experiments in production, lower environments such as dev, QA, and performance environments can be your playground for your chaos experiments. Often, most of the organization will have more than one account, just to isolate the engineering units,  environments, and more, just to avoid stepping on each other's toe. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>As a chaos engineer, you may need to run experiments on the applications which are hosted in different AWS accounts. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Consider a simple application hosted on an EC2 instance in the AWS Account <code>marketing-team</code>. LitmusChaos on AWS EKS runs on a<code> performance-team</code>  AWS account. In this case, we need to enable cross - AWS account access.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9454,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-32.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-32-1024x381.png" alt="Chaos Engineering with LitmusChaos on AWS -  Cross Account Implementation" class="wp-image-9454"/></a><figcaption>Chaos Engineering with LitmusChaos on AWS - Cross Account Implementation</figcaption></figure>
<!-- /wp:image -->

<!-- wp:generateblocks/headline {"uniqueId":"3bd0a0a9"} -->
<h2 class="gb-headline gb-headline-3bd0a0a9 gb-headline-text">Prerequisites</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>The following are the prerequisites for implementing this setup:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Two AWS Accounts with IAM privileges</li><li>LitmusChaos on AWS Account A</li><li>Sample application on AWS Account B</li><li>eksctl</li><li>kubectl</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Creating two AWS accounts might be tricky as you need two different credit cards to sign up. I have borrowed an account from my friend, just to demonstrate this implementation.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Also, make sure you have valid privileges to play around with the IAM, Policy, Trust Relationships and more.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let us name our AWS accounts to understand better. You are a Chaos Engineer who provides Chaos-as-a-Service. You run LitmusChaos on your AWS account called <code>Chaos-Account</code>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>A customer has approached you to perform chaos experiments on their application. Let us call their AWS account a <code>Customer-Account</code>.  </p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"24e33c81"} -->
<h2 class="gb-headline gb-headline-24e33c81 gb-headline-text">On Chaos-Account</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>On <code>Chaos-Account</code> make sure you have LitmusChaos is up and running successfully and able to run experiments either using Self-Agent or external agent.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Create a new IAM Role using the below policy and trust relationship.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Before creating IAM Role, make sure you have the valid OIDC provider URL.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To validate OIDC URL, enter the below command</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>aws iam list-open-id-connect-providers</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The above command must return the OIDC entry e.g. oidc.eks.us-east-2.amazonaws.com/id/F25CB34E2ACAA3CE669BA4E0XXXXXXXX"</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If not, create an OIDC using the below command. In this case, our cluster name is <code>chaos-cluster</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>eksctl utils associate-iam-oidc-provider --cluster chaos-account --approve</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Create a policy <code>Chaos-Account-Policy</code> using the below JSON. Please follow the Zero Trust framework while creating a policy and trust relationship. I am just being playful as it is my demo account.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": &#91;
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:*",
            "Resource": &#91;
                "*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sts:*",
            "Resource": &#91;
                "*"
            ]
        }
    ]
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Create a new IAM role called <code>Chaos-Account-Role</code> and attach the above policy and add the below trust relationship. Make sure you replace the OIDC ID, account ID, region etc.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": &#91;
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::&lt;account_id&gt;:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/F25CB34E2ACAA3CE669BA4EXXXXXXX"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/F25CB34E2ACAA3CE669BA4EXXXXXXX:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}</code></pre>
<!-- /wp:code -->

<!-- wp:generateblocks/headline {"uniqueId":"00d10d91"} -->
<h2 class="gb-headline gb-headline-00d10d91 gb-headline-text">On <code>Customer-Account</code></h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Login into your <code>Customer-Account</code> or ask them to create a new policy called <code>Customer-Account-Policy</code> using the below JSON. Again, please follow the Zero Trust policy.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": &#91;
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": &#91;
                "ec2:*",
                "sts:*"
            ],
            "Resource": "*"
        }
    ]
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Before creating a new role, let us create Identity provider in this account. Go to <strong>IAM</strong>, under <strong>Access management</strong>, click <strong>Identity providers</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Then, click <strong>Add provider</strong> button.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Select <strong>OpenID Connect</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In <strong>Provider URL</strong> text box, enter the <code>Chaos-Account</code> identity provider URL, and then click <strong>Get thumbprint</strong> button.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In <strong>Audience </strong>text box, enter <code>sts.amazonaws.com</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Click on <strong>Add provider</strong> button.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9457,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/QAInsights-NaveenKumar-00012.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/QAInsights-NaveenKumar-00012.png" alt="Add Provider" class="wp-image-9457"/></a><figcaption>Add Provider</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Create a new role by selecting <strong>Web identity </strong>as shown below and select your <strong>Identity provider</strong> and <strong>Audience </strong>[i.e. <code>Chaos-Account</code>] and then click <strong>Next</strong></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9461,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-35.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-35-1024x402.png" alt="Adding Web Identity Role" class="wp-image-9461"/></a><figcaption>Adding Web Identity Role</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>In the next page, select the <code>Customer-Account-Policy</code> then create a role <code>Customer-Account-Role</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9462,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-36.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-36-1024x408.png" alt="Customer Account Role" class="wp-image-9462"/></a><figcaption>Customer Account Role</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Copy the <code>Customer-Account-Role</code> ARN to the clipboard as we need it to annotate in the <code>LitmusChaos</code> service account.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"ea31095d"} -->
<h2 class="gb-headline gb-headline-ea31095d gb-headline-text">Annotation</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Use the below command to annotate your LitmusChaos service account in the default namespace <code>litmus</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>kubectl annotate serviceaccount -n litmus litmus-admin eks.amazonaws.com/role-arn=arn:aws:iam::&lt;customer-account-id&gt;:role/Customer-Account-Role --overwrite</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To validate the annotation, use the below command:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>kubectl describe sa -n litmus litmus-admin</code></p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"9c71ac7b"} -->
<h2 class="gb-headline gb-headline-9c71ac7b gb-headline-text">Chaos Experiment</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Now it is time to create a chaos experiment to stop the EC2 instance in <code>Customer-Account</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Copy the instance ID and the region which we need to feed it in the experiment YAML.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Create a <code>ec2-terminate-by-id</code> experiment and configure the instance ID and the region. Make sure you remove the secrets from the YAML and schedule it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If all is well, it should stop and start the EC2 instance in your customer account.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9463,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-37.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-37-1024x384.png" alt="Graphical View of Chaos Result" class="wp-image-9463"/></a><figcaption>Graphical View of Chaos Result</figcaption></figure>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":9464,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-38.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-38.png" alt="Chaos Verdict" class="wp-image-9464"/></a><figcaption>Chaos Verdict</figcaption></figure>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":9465,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/05/image-39.png"><img src="https://qainsights.com/wp-content/uploads/2022/05/image-39-1024x314.png" alt="" class="wp-image-9465"/></a><figcaption>Pod Logs</figcaption></figure>
<!-- /wp:image -->

<!-- wp:generateblocks/headline {"uniqueId":"d0b1ad71"} -->
<h2 class="gb-headline gb-headline-d0b1ad71 gb-headline-text">Conclusion</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>As you learned, cross - account implementation is easy and straightforward. It looks complex, but once you understand the OIDC concept, how role, policy, and trust relationships works, it is quick to implement. The above steps can be automated by leveraging AWS CLI and eksctl. </p>
<!-- /wp:paragraph -->