Performance Engineering - The Reliability Edition
Tags: performance, testing, devops, sre
Published: 2021-03-22 19:59:05

---
title: 'Performance Engineering - The Reliability Edition'
published: true
summary: '[Tech] Better understand and define Reliability through performance methodologies and principles'
---
#### **Question**

_Can we improve the reliability of a system by employing various performance engineering techniques to different stages of the development process?_

> This is a look at how a solid **Performance Engineering** strategy that uses **Reliability** principles and DevOps idealisms to complement and strengthen current or proposed performance initiatives

> These approaches attempt to achieve better business cohesion, reliability and velocity benefits. To do this we can look at applying various methodologies from [Performance Engineering](#the-performance-engineering-model) using a [Shift left](#shift-left) and [Move Right](#move-right) approaches that extend [Traditional Performance Testing](#traditional-performance-testing) techniques

##### **At its core, to understand an applications performance we need**

- A mechanism to run load against an application or system
- A way of measuring how they performed
- A way of comparing the results against what we believe is the ideal state

Each area of performance within the DevOps model has its part to play. That is, they all relate in some shape or form to the principles around building, defining and maintaining a reliable system

## In a nutshell

Each Performance execution and analysis piece should look to be guided by the Engineering Efficiency, DevOps and Reliability principles that apply to software development

[![The Breakdown](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ztl2qcj3glhwok0dse7p.png)](https://photos.app.goo.gl/beByCbJbjipV3cZUA)

- **Reliability Engineering(RE)** attempts to predict and prevent the risk of there being a failure whether that be a component or an entire system of services

- **Performance Engineering(PE)** states we should start earlier in the [SDLC](https://en.wikipedia.org/wiki/Systems_development_life_cycle) to get faster feedback, but also extends into Operations and Support to use real world data to build/update of the performance models (scripts and analysis)

- **Performance Testing (PT**) is all about determining what the performance of an application is (**baselining**) or comparing to how you believe it should be(**delta analysis**) under various conditions and situations in the 'test' environment

## **A look at Performance Engineering**

PE looks incorporate the methodologies of **'Agile'** and use these in conjunction with **'DevOps'** idealisms in order to provide a improved approach that adds value rather than one that tends to hinder delivery velocity

We can do this by looking at adopting a left shift / move right approach that incorporates a cloud first performance automation approach. This can then lead to reduced feedback cycle (**velocity increase**) and bottlenecks / bugs being caught early on (**reliability increase**).

## **The Performance Engineering Model**

**PE** is all about applying process and strategies at each step of the SDLC, the following are example actions/options that can be applied within each vertical

[![PE Model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sue8bc6h6rpxs4umsvou.png)](https://photos.app.goo.gl/86LeiNxjVgrGFtci7)

_The idea being that performance is a consideration at each step in the software lifecycle, The captured metrics are gathered from Dev, Test, Deploy and Operations and used to refine the next cycle of performance_

### **Traditional performance testing**

Quite often done within the [test phase](https://en.wikipedia.org/wiki/Systems_development_life_cycle#Testing) and entails a big bang approach that consists of many pods/VM's to generate load against an application/system

[![PE Traditional](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6gi969uac77757b4sgqb.png)](https://photos.app.goo.gl/8tJ2BFpGWGiPV5EBA)

| **Pro's**                                                        |                            **Con's**                            |
| :--------------------------------------------------------------- | :-------------------------------------------------------------: |
| **Simulates real world conditions as closely as possible**       | Often a integrated(shared) environment which can affect results |
| **Integrated tests execute against multiple components at once** | Data is often 'test' data which could affect behaviour/results  |
| **Tools can replicate thousands (if not more) of users**         |        Replicating 'Prod' environments can be expensive         |
| **Extensive metrics/reports from tool**                          |    Finding route case when diagnosing issues can be complex     |
|                                                                  |       Commercial Tooling can be expensive to operate item       |

---

## Performance/Reliability options to improve efficiencies, engagement and observability

---

--> We can attempt to find this out using combination PE, RE and DevOps principles and methodologies

## **Shift Left Approach**

Reducing the SDLC feedback loop to uncover and rectify potential system and environment issues early

[![Shift Left](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bz390w39ku55odbzl7ji.png)](https://photos.app.goo.gl/megEUntMiiqo2jFX9)

### **Shift Left Benefits**

|                        Item |                                                Description |
| --------------------------: | ---------------------------------------------------------: |
|           **Team cohesion** |              Foster developer engagement and contribution. |
|               **Less bugs** |                                 Reduced development costs. |
|    **Improved performance** |            Detect and eliminate bottlenecks shortly early. |
|            **Reduced risk** |                  Find bugs and performance issues earlier. |
| **Speed up time-to-market** | Having more trust in your applications and infrastructure. |

## **Move Right Approach**

A "Move Right" approach extends testing out to include user feedback and metrics from your production environment. This can then be used to update the performance model that's developed as a consequence

[![Move Right](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1356707is62op9ul0xlg.png)](https://photos.app.goo.gl/fZR7igu1XReeZioW7)

### **Move Right Benefits**

|                                       Item |                                                                                  Description |
| -----------------------------------------: | -------------------------------------------------------------------------------------------: |
|              **Increased User experience** |                                        Tests closer match the actions expected by your users |
|                      **Responding faster** | Teams have more involvement and ownership over the performance information is presented back |
|            **Design hypothesis evaluated** |                               Assumption are reflected upon and adequate action can be taken |
| **Various performance management options** |       Many different tools for being able to change traffic flows that can alter performance |

### **Measurements and Observability**

The use of performance metrics from each environment (**Dev/Test/Prod**) are used to determine whether they are within [SLO's](https://sre.google/resources/practices-and-processes/art-of-slos/) limits.

Idea being we can understand and easily record local (**component**) and integrated(**end 2 end**) metrics to provide better performance transparency. These then would be compared to ideal state

These SLO's can be enforced through the use of [SLI's](https://sre.google/workbook/implementing-slos/) (SLI specifications and SLI implementations) and compared to our error budget to measure tolerance

[![Observability](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/98fetiakmfrpg0fghkm8.png)](https://photos.app.goo.gl/zwah66Awbw1zfqZ9A)

_With the view to obtain an current state view of our applications performance in each environment and at each stage of the SDLC these are then compared against our business performance exceptions defined in the SLO and enforced in the SLI_

###### **Performance SLI implementations could include:**

- API / UI response times
- DB transaction times
- Pod / VM scaling events
- CPU use / Network activity / Memory usage

Could all be defined and compared using SLI's

A subset of the performance suite can be used to **poke** test (performance smoke test) the application after deployment. A degraded Performance run could then trigger a rollback

### _Summary_

> A balanced performance strategy that is applied at each stage of the SDLC, that uses guidance from RE principles provides a more well rounded verification process and in turn lead to a culture of empathy, encourage collaboration, reduce delivery cycle duration and mitigate the chance of deploying underperforming software