It's all Chaos! And it Makes for Resilience at Scale
Tags: sre, devops
Published: 2021-03-15 16:16:45

By: Emily Arnott

Originally published on [Failure is Inevitable](https://www.blameless.com/blog/chaos-engineering-makes-for-resilience-at-scale).

Chaos engineering is a practice where engineers simulate failure to see how systems respond. This helps teams proactively identify and fix preventable issues. It also helps teams prepare responses to the types of issues they cannot prevent, such as sudden hardware failure. The goal of chaos engineering is to improve the reliability and resilience of a system. As such, it is an essential part of a mature SRE solution.

But integrating chaos engineering with other SRE tools and practices can be challenging. To get the most from your experiments, you’ll need to tie in learnings across all your reliability practices. You’ll also need to adjust your chaos engineering as your organization scales. In this blog post, we’ll look at:
* How SRE and chaos engineering intersect
* Best practices for chaos engineering
* What experiments look like at different maturity levels

#Chaos engineering and SRE#
It’s clear how the goals of SRE and chaos engineering align. Both practices encourage teams to build resilience into their systems. But the connections don’t stop there. Many SRE practices integrate with chaos engineering to increase the effectiveness of both. Below are a few examples.

**SLOs as chaos engineering scoreboards**
When running chaos engineering experiments, it's important to determine how impactful the hypothetical failure would be. This can be difficult.

Consider a test that shows that an entire service would go offline if a certain server fails. You estimate that it would take an hour or so to return to normal operation. It sounds frightening, but what if that service was only used by a tiny fraction of your customers?

Another test might show that, when traffic surpasses a certain threshold, a page accessed by every customer loads 3 seconds slower. This scenario could have more customer impact than the other. Teams would want to focus on resolving this issue first.

SLOs allows you to compare these scenarios using the most important metric: customer impact. SLIs, or [service level indicators](https://www.blameless.com/blog/slis-understand-users-needs), are built from the service metrics that matter most to customers. SLOs, or [service level objectives](https://www.blameless.com/blog/service-level-objectives-slos-lessons-learned), show the level of failure customers will tolerate.

When you run chaos experiments, you can determine how the experiment would affect the SLO. This gives you a triaging model for the lessons of different experiments. You can then focus on preventing the incidents that affected the SLO most.

**Chaos engineering as runbook bootcamp**
It's important to simulate the impact of a hypothetical failure and work to prevent it. But there will still be incidents, no matter how many experiments we run. Chaos engineering also gives teams the space to practice response measures. This can help responders work faster and with more confidence during a real incident.

In SRE, incident responses are codified as [runbooks](https://www.blameless.com/blog/introducing-blameless-runbook-documentation). These are guides broken down into modular checks and steps. Where possible, [runbooks are automated](https://www.blameless.com/blog/runbook-automation-best-practices) to save toil. Of course, runbooks can never be perfect. Regular review is necessary to ensure that all information is up-to-date and comprehensive.

Chaos engineering can help improve runbooks by providing more opportunities to evaluate them. Teams will not use runbooks addressing a type of rare, catastrophic failure often. When the failure does occur, your team will need to know it can trust the runbook. By running chaos experiments of this scenario, you’ll find potential stumbling blocks.

Runbooks can also serve as inspiration for chaos experiments. If a runbook has been “gathering dust,” you can design an experiment to put it to use. This will ensure it’s up-to-date with your system and still useful.

**Building a library of chaos engineering retrospectives**
A valuable tool in your SRE tool belt is the [incident retrospective](https://www.blameless.com/blog/incident-retrospective-postmortem-template). This is a document built by teams responding to an incident. It contains the incident timeline, key communications, follow-up actions, and more. Incident retrospectives form a valuable hub of knowledge. They are invaluable for [onboarding](https://www.blameless.com/blog/4-ways-sre-helps-new-employees-onboard) and developing a culture of [continuous improvement](https://www.blameless.com/resources/building-a-culture-of-continuous-improvement).


Chaos engineering can help build your library of retrospectives. Teams should write retrospectives about chaos experiments as they would for a real incident. Include details about why and how the experiment was conducted to be thorough. Reviewing these retrospectives can provide the same beneficial insights as a real incident.

Conversely, incident retrospectives can motivate good chaos experiments. Imagine your team had difficulty responding to a particular incident. Reviewing the incident retrospective revealed why the team stumbled. Your team creates a plan for incidents like these moving forward. “Replaying” the incident will give you a direct comparison between the new and old methods. It can help you avoid making the same mistakes.

#A maturity model of chaos engineering#
As organizations grow in maturity, adopting chaos engineering as a practice provides more opportunities. But there are also challenges. This chart breaks down what to expect at each stage.

![Table sharing chaos engineering challenges and opportunities for beginning, intermediate, and advanced teams.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/db5f7970qhs9z0mw1js2.png)

No matter what maturity your organization is, the best time to try chaos engineering is now. The sooner you can build experimenting into your routines, the more time you’ll have to develop your expertise.

Blameless can help you make the most of your chaos engineering experiments. Our SLO, runbook documentation, and incident retrospective tools can help you get the most from every experiment. To see how, check out a [demo](https://www.blameless.com/schedule-demo).

If you enjoyed this blog post, check out these resources:
* [eBook: The Complete Guide to Pragmatic Incident Response](https://www.blameless.com/resources/the-complete-guide-to-pragmatic-incident-response)
* [Podcast: Tammy Bryant and Eric Roberts The Importance of Glue Work](https://www.blameless.com/blog/resilience-in-action-episode-5)
* [SRE Leaders Panel: Testing In Production](https://www.blameless.com/resources/sre-leaders-panel-testing-in-production)