Getting Started with Litmus SDK
Tags: kubernetes, docker, litmuschaos, chaosengineering
Published: 2020-06-16 10:38:00

In this blog, I will talk about the generation of custom/new chaos experiment with the help of <a href="https://github.com/litmuschaos/litmus-go"> Litmus SDK </a>. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing Chaos Engineering in <a href="https://dev.to/umamukkara/chaos-engineering-for-cloud-native-systems-2fjn"> cloud-native environments</a>. Litmus provides a chaos-operator, a large set of chaos experiments in its <a href="https://hub.litmuschaos.io"> HUB</a>, detailed <a href="https://docs.litmuschaos.io/docs/getstarted/"> documentaion</a>, quick <a href="https://dev.to/uditgaurav/get-started-with-litmuschaos-in-minutes-4ke1"> demo</a>, and a friendly <a href="https://litmuschaos.io">community</a>.
![Litmus Flow Diagram](https://dev-to-uploads.s3.amazonaws.com/i/cdmbnz9g0200z0geqp8f.png)

<h1> What is Litmus SDK? </h1>

The Litmus SDK provides a simple way to bootstrap your experiment and helps create the aforementioned artifacts in the appropriate directory (i.e., as per the chaos-category) based on an attributes file provided as input by the chart-developer. The scaffolded files consist of placeholders which can then be filled as desired.

It generates the custom chaos experiments with some default Pre & Post Chaos Checks (AUT & Auxiliary Applications status checks). It can use the existing chaoslib (present inside `/chaoslib` directory), if available else It will create a new chaoslib inside the corresponding directory.

<h2> Life Cycle of a Chaos Experiment</h2>
 Each Chaos Experiment is divided into six main sections:

- <b>Prepare:</b> Prepare section contains the initialization steps (get ENVs), Updation of chaos result to marked the beginning of the chaos experiment.
- <b>PreChaosCheck:</b> PreChaosCheck contains some default checks (AUT & Auxiliary Application status check) and custom checks, which may vary on the basis of experiment i.e, liveness check, data persistence check, etc.
- <b> ChaosInject:</b><i> ChaosInject invoke the actual chaos, It contain the main business logic.
- <b> CleanUp:</b> CleanUp contains the steps to remove the helper/external pod, if any</i>
- <b>PostChaosCheck:</b> PostChaosCheck contains the similar steps as the PreCheckCheck section have. It used to ensure the resiliency after chaos injection.
- <b> Summary </b> Summary updates the verdict(Pass/Fail) inside the chaos result and FailStep if the experiment fails.


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/t89sd1fyaabm2jqeqju7.png)

<h2>Pre-requisites</h2>

<h3>Go Experiments</h3>
- go should be is available & the GOPATH env is configured appropriately

<h3>Ansible Experiments</h3>
- python3 is available (sudo apt-get install python3.6)
- jinja2 & pyYaml python packages are available (sudo apt-get install python3-pip, pip install jinja2, pip install pyYaml)

<h2>Steps to Generate Experiment via Litmus SDK</h2>
 - Clone the [litmus-go](https://github.com/litmuschaos/litmus-go) repository for go experiments and [litmus-ansible](https://github.com/litmuschaos/litmus-ansible) for the ansible experiments and navigate to the contribute/developer-guide folder.

```yaml
## for litmus-go
$ git clone https://github.com/litmuschaos/litmus-go.git
$ cd litmus-go/contribute/developer-guide
```
```yaml
## for litmus-ansible
$ git clone https://github.com/litmuschaos/litmus-ansible.git
$ cd litmus-ansible/contribute/developer_guide
```
- Populate the attributes.yaml with details of the chaos experiment (or chart). Use the [attributes.yaml.sample](https://github.com/litmuschaos/litmus-go/blob/master/contribute/developer-guide/attributes.yaml.sample) as reference.

As an example, let us consider an experiment to kill one of the replicas of an Nginx deployment. The `attributes.yaml` can be constructed like this:
```yaml
$ cat attributes.yaml 

---
name: "sample-pod-delete"
version: "0.1.0"
category: "sample-category"
repository: "https://github.com/litmuschaos/litmus-go/tree/master/sample-category/pod-delete"
community: "https://kubernetes.slack.com/messages/CNXNB0ZTN"
description: "kills nginx pods in a random manner"
keywords:
  - "pods"
  - "kubernetes"
  - "sample-category"
  - "nginx"
scope: "Namespaced"
auxiliaryappcheck: false
permissions:
  - apigroups:
      - ""
      - "batch"
      - "litmuschaos.io"
    resources:
      - "jobs"
      - "pods"
      - "chaosengines"
      - "chaosexperiments"
      - "chaosresults"
    verbs:
      - "create"
      - "list"
      - "get"
      - "update"
      - "patch"
      - "delete"
maturity: "alpha"
maintainers:
  - name: "ispeakc0de"
    email: "shubham.chaudhary@harness.io"
provider:
  name: "Harness"
minkubernetesversion: "1.12.0"
references:
  - name: Documentation
    url: "https://docs.litmuschaos.io/docs/getstarted/"
```
- Run the following command to generate the necessary artifacts for submitting the sample-category chaos chart with pod-delete experiment.
```yaml
## litmus-go
$ ./litmus-sdk generate <generate-type> -f=attributes.yaml
```

- <b>Note:</b> Replace the `<generate-type>`placeholder with the appropriate value based on the usecase:
    - `experiment:` Chaos experiment artifacts belonging to a an existing OR new chart.
      - Provide the type of chaoslib in the `-t` flag. It supports the following values
         - `exec`: It creates the exec based chaoslib(default type)
         - `helper`: It creates the helper based chaoslib
    - `chart:` Just the chaos-chart metadata, i.e., chartserviceversion yaml
      - Provide the type of chart in the `-t` flag. It supports the following values
         - `category`: It creates the chart metadata for the category i.e chartserviceversion, package manifests
         - `experiment`: It creates the chart for the experiment i.e chartserviceversion, engine, rbac, experiment manifests
         - `all`: it creates both category and experiment charts (default type)
    - provide the path of the `attribute.yaml` manifest in the `-f` flag.

```yaml
## litmus-ansible
$ python3 generate_chart.py --attributes_file=attributes.yaml --generate_type=experiment
```
- <b>Note:</b> In the `--generate_type` flag, select the appropriate type of manifests to be generated, where,
    - `chart:` Just the chaos-chart metadata, i.e, chartserviceversion yaml
    - `experiment:` Chaos experiment artifacts belonging to an existing or new chart.
    - provide the path of the `attribute.yaml` manifest in the `--attributes_file` flag.


<h2>Verify the Generated Files</h2>
```yaml
$ cd /experiments

$ ls -ltr

total 8
drwxr-xr-x 3 shubham shubham 4096 May 15 12:02 generic/
drwxr-xr-x 3 shubham shubham 4096 May 15 13:26 sample-category/


$ ls -ltr sample-category/

total 12
-rw-r--r-- 1 shubham shubham   41 May 15 13:26 sample-category.package.yaml
-rw-r--r-- 1 shubham shubham  734 May 15 13:26 sample-category.chartserviceversion.yaml
drwxr-xr-x 2 shubham shubham 4096 May 15 13:26 sample-pod-delete/

$ ls -ltr sample-category/sample-pod-delete

total 28
-rw-r--r-- 1 shubham shubham  791 May 15 13:26 rbac.yaml
-rw-r--r-- 1 shubham shubham  734 May 15 13:26 sample-pod-delete.chartserviceversion.yaml
-rw-r--r-- 1 shubham shubham  792 May 15 13:26 experiment.yaml
-rw-r--r-- 1 shubham shubham  813 May 15 13:26 engine.yaml
## this file will be created in case of litmus-go
--rw-r--r-- 1 shubham shubham 4096 May 15 13:26 test
-rw-r--r-- 1 shubham shubham 4533 May 15 13:26 sample-pod-delete.go
## this file will be created in case of litmus-ansible
-rw-r--r-- 1 shubham shubham 1777 May 15 13:26 pod-delete-k8s-job.yml
-rw-r--r-- 1 shubham shubham 4533 May 15 13:26 pod-delete-ansible-logic.yml

$ ls -ltr sample-category/sample-pod-delete/test

total 4
-rw-r--r-- 1 shubham shubham  1039 May 15 13:26 test.yaml
```
- Proceed with construction of business logic inside the `pod-delete.go` file in litmus-go or pod-delete-ansible-logic.yml in litmus-ansible, by making the appropriate modifications listed below to achieve the desired effect:

   - <i>variables</i>
   - <i>entry & exit criteria checks for the experiment</i>
   - <i>helper utils in either pkg or new base chaos libraries</i>

- Update the `experiment.yaml` with the right chaos params in the `spec.definition.env` with their default values.

- Update the `chaoslib/litmus/sample-pod-delete/lib/pod-delete.go` chaoslib to achieve the desired effect or reuse the existing chaoslib.

- Create an experiment README explaining, briefly, the what, why & how of the experiment to aid users of this experiment.

<h2>Steps to Test Experiment</h2>
Follow the [Litmus SDK: DevTest your chaos experiments with Okteto](https://dev.to/ksatchit/litmus-sdk-devtest-your-chaos-experiments-with-okteto-4dkj) blog to test the generated experiment.

<h3>Steps to Include the Chaos Charts/Experiments into the ChartHub</h3>
- Send a PR to the `litmus-go/litmus-go` repo with the modified experiment files i.e, [pod-memory-hog](https://github.com/litmuschaos/litmus-go/pull/25)
- Send a PR to the chaos-charts repo with the modified experiment CR, experiment chartserviceversion, chaos chart (category-level) chartserviceversion & package (if applicable) YAMLs i.e, [kubelet service kill](https://github.com/litmuschaos/chaos-charts/pull/239)

<h2>Conclusion</h2>

Litmus SDK will help developers & SREs to create their custom chaos experiment on demand. It will generate the templates, pre/post chaos checks, chaoslib (which can be modified according to use-case).

<i>Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you? Join Our Community #litmus channel in [Kubernetes Slack](https://kubernetes.slack.com/messages/CNXNB0ZTN)
Contribute to LitmusChaos and share your feedback on [Github](https://github.com/litmuschaos/litmus)
If you like LitmusChaos, become one of the many stargazers [here](https://github.com/litmuschaos/litmus/stargazers){% github litmuschaos/litmus %}</i>