A little bit of chaos
Tags: chaosengineering
Published: 2022-02-03 21:33:09

---
title: A little bit of chaos
published: true
date: 2022-01-03 00:00:00 UTC
tags: chaosengineering
canonical_url: https://jhall.io/archive/2022/01/03/a-little-bit-of-chaos/
---

If you’re not familiar with the concept of [chaos engineering](https://en.wikipedia.org/wiki/Chaos_engineering), you should be.

Often conflated with [Chaos Monkey](https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey), which is just one high-profile early example of the practice, chaos engineering is the practice of _intentionally_ breaking things in production to build confidence in a system’s ability to withstand unexpected conditions.

As explained in the book [_Chaos Monkeys_](https://amzn.to/3ES5X9A) (the book really has little to do with engineering, but it’s still an entertaining read):

> Imagine a monkey entering a ‘data center’, these ‘farms’ of servers that host all the critical functions of our online activities. The monkey randomly rips cables, destroys devices and returns everything that passes by the hand [i.e. flings excrement]. The challenge for IT managers is to design the information system they are responsible for so that it can work despite these monkeys, which no one ever knows when they arrive and what they will destroy.

How would _your_ system behave if a monkey entered your data center and did those things?

Wouldn’t it be great if your system continued to operate, or at the very least, informed you of the problems in time to fix them, under the influence of such chaos?

If you’re hosting your service in the cloud, the big cloud providers actually give you a really easy\* way to introduce a little bit of chaos into your system, and actually _save money_ at the same time.

What’s the trick? Unstable VMs!

Amazon offers [EC2 Spot Instances](https://aws.amazon.com/ec2/spot/), Google offers (Preemptible VM instances)([https://cloud.google.com/compute/docs/instances/preemptible),](https://cloud.google.com/compute/docs/instances/preemptible%29,) and Azure has its [Spot VMs](https://azure.microsoft.com/en-us/services/virtual-machines/spot/).

The concept is simple: You get a discount on the normal VM fees in exchange for practically _no_ stability guarantee. In fact, in some configurations, you’re guaranteed that your VM will live at most 24 hours.

Aside from saving money, why would this be hepful?

Well, if you want to be sure that your application can withstand random restarts and crashes, how better to test that capability than to host it entirely on Spot/Preemptible VMs, that will force a restart potentially every few hours?

To be clear, this is _not_ a full-fledged replacement for proper chaos engineering. It’s a baby step. But it’s an easy\* one.

* * *

\*Easy, you say? Well, setting up Spot/preemptible VMs is easy. You can do that in just a few minutes. The hard part comes when you discover that your application doesn’t work well when it’s being constantly restarted or goes down sometimes. But then again, that is the point of the exercise, right?

* * *
_If you enjoyed this message, [subscribe](https://jhall.io/daily) to <u>The Daily Commit</u> to get future messages to your inbox._