Getting Started with Chaos Engineering
Tags: beginners, chaos, testing
Published: 2022-06-30 15:07:59

<!-- wp:paragraph -->
<p>Breaking stuff on purpose primarily in the production environment is one of the mantras in Chaos Engineering. But when you tell your plan to your engineering manager or product owner, often we get some resistance. Their concerns are valid, what-if breaking stuff is irreversible, what will happen to the end-users, will our support ticket system get busy and more. This blog post will help you get started with Chaos Engineering practice in your organization.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"1fec561b"} -->
<h2 class="gb-headline gb-headline-1fec561b gb-headline-text">What is Chaos Engineering?</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>There are various definitions available from industry leaders about chaos engineering. Here is my slide from one of my videos :</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9521,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/06/image-17.png"><img src="https://qainsights.com/wp-content/uploads/2022/06/image-17-1024x446.png" alt="Chaos Engineering Definitions" class="wp-image-9521"/></a><figcaption>Chaos Engineering Definitions</figcaption></figure>
<!-- /wp:image -->

<!-- wp:embed {"url":"https://youtu.be/8qib94Gh1ws","type":"video","providerNameSlug":"youtube","responsive":true,"className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://youtu.be/8qib94Gh1ws
</div></figure>
<!-- /wp:embed -->

<!-- wp:generateblocks/headline {"uniqueId":"a45f6d39"} -->
<h2 class="gb-headline gb-headline-a45f6d39 gb-headline-text">Getting Started</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>The goal of chaos experiments is to learn how our system will behave in case of catastrophic failures in production, how resilient our system is which gives us an opportunity to optimize and fix the issues. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here is how you can begin your chaos engineering practice.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"95b8175e"} -->
<h2 class="gb-headline gb-headline-95b8175e gb-headline-text">Get a buy-in from your manager</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>The first step is to get an approval from your manager to carry out the experiments in the test environment. Typically, chaos experiments should be carried out in production. But I suggest you take baby steps. You can perform chaos experiments in any valid environment. If the production environment is not available, I recommend running the experiments in a non-production (or stage) environment. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Explain the values that you are bringing by performing the chaos experiments, see below to name a few :</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Identifying failures and bottlenecks</li><li>Resilient validation</li><li>Scaling validation</li></ul>
<!-- /wp:list -->

<!-- wp:generateblocks/headline {"uniqueId":"e7aeb168"} -->
<h2 class="gb-headline gb-headline-e7aeb168 gb-headline-text">Understand the Architecture</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>System fails, all the time. Before running your chaos experiments, thoroughly understand your system's architecture. Have a working session with your developers, architects, SREs and brainstorm the application architecture and understand the upstream / downstream components, dependencies, timeline, game day schedule, and more. This will help you to understand where your system could fail. </p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"5a830cef"} -->
<h2 class="gb-headline gb-headline-5a830cef gb-headline-text">Write Hypothesis</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Start writing the list of hypothesis (/h??päTH?s?s/) (<em>a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation).</em></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>e.g. given a Kubernetes deployment, deleting one pod should not increase the service response time under the typical load.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Another example : load balancer must route the requests only to the healthy and running nodes.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There is no right or wrong while writing a hypothesis. It is an iterative process.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Our objective is NOT to make our hypothesis PASS or FAIL. Each hypothesis will give us an opportunity to understand our system.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"a901bf5a"} -->
<h2 class="gb-headline gb-headline-a901bf5a gb-headline-text">Minimize the Blast Radius </h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Always start small. Reduce the impact on the end users while running the chaos experiments by minimizing the blast radius. E.g. instead of deleting a deployment in the Kubernetes cluster, delete the pods and validate the resiliency. Even if you are deleting a deployment, make sure GitOps is in effect, so that GitOps flow will create a deployment automatically. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Another example: instead of shutting down all the nodes in the cluster, go for 50% of running nodes or instead of plugging off the entire region, shutdown a zone. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Once the chaos process is matured and your team is in a comfortable zone, you can slowly increase the blast radius. </p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/image {"uniqueId":"5d863644","mediaId":9525,"sizeSlug":"full","objectFit":"inherit"} -->
<figure class="gb-block-image gb-block-image-5d863644"><img class="gb-image gb-image-5d863644" src="https://qainsights.com/wp-content/uploads/2022/06/Checkmark.jpeg" alt="Getting Started with Chaos Engineering" title="Checkmark"/><!-- wp:generateblocks/headline {"uniqueId":"993e7981","element":"figcaption","alignment":"center","isCaption":true} -->
<figcaption class="gb-headline gb-headline-993e7981 gb-headline-text">Getting Started with Chaos Engineering</figcaption>
<!-- /wp:generateblocks/headline --></figure>
<!-- /wp:generateblocks/image -->

<!-- wp:generateblocks/headline {"uniqueId":"37b83f59"} -->
<h2 class="gb-headline gb-headline-37b83f59 gb-headline-text">Plan for a Game Day</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Plan ahead and always have a Plan B. Notify all your stakeholders at least one week before and start a unified communication channel in Slack (or your company's collaboration platform) to post updates regularly. I suggest having your developers, or DevOps or SRE team when you run your first experiment.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"7932c5b6"} -->
<h2 class="gb-headline gb-headline-7932c5b6 gb-headline-text">Run your First Experiment</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>No one is perfect. It is okay if you are having trouble running your first experiment. Post an update promptly and inform all the stakeholders.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you are able to run your first experiment successfully, here is my virtual pat. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Believe me, running the first chaos experiment is like riding a high thrill level roller coaster. If things go south, make sure you are able to halt the experiment and revert the infrastructure with the help of your DevOps or SRE team.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>During the experiment, monitor your Observability dashboard and observe the vitals such as response time, disk utilization, pass / fail transactions, health checks, and more.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"cc0cde2d"} -->
<h2 class="gb-headline gb-headline-cc0cde2d gb-headline-text">Analysis</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Once the experiment is done, log all your observations in a spreadsheet, analyze them and define your hypothesis verdict. Again, there is no pass or fail; it is a learning.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"c8ff279f"} -->
<h2 class="gb-headline gb-headline-c8ff279f gb-headline-text">Brainstorm</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Schedule a meeting with your developers, architects, DevOps / SRE team to discuss your verdict. This will help the team to understand the verdicts and fix the issues which you discovered. Once the issues are addressed, you can rerun the experiments.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you find that the system is resilient, you can try increasing the blast radius and rerun the experiments.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"18e63aa3"} -->
<h2 class="gb-headline gb-headline-18e63aa3 gb-headline-text">Next Steps</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>After running various game days, you can learn about team dynamics, system performance, and more. The next step is embedding chaos experiments into your developer workflow, so that chaos experiments will be automated, which will bring more confidence to your team. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Happy Chaos'ing!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>  </p>
<!-- /wp:paragraph -->