How do you know you are doing DevOps right?
Tags: devops, agile
Published: 2021-04-12 12:55:48

In refining a DevOps process, it can be difficult to determine if a change had a positive effect.  Continuously measuring and analyzing metrics can show a change is moving the system in the right direction.  Before you go off and start counting lines-of-code or hours spent, let's talk about meaningful metrics.

Back in 2019, Google and few other heavy hitters put together a study detailing measurements high performing organizations shared.  The results are available in the [State of DevOps](https://services.google.com/fh/files/misc/state-of-devops-2019.pdf).  It is worth the read, but if you need a good summary of the metrics, my co-worker has put one together [here at Dev.to](https://dev.to/leading-edje/measuring-your-team-s-software-delivery-performance-4795).  While the metrics themselves are interesting, the reasons for them are just as intriguing.  The four metrics are as follows:
* _**Lead Time for Change**_ : throughput of software delivery process from check-in to release
* _**Deployment Frequency**_ : how often code is released to production
* _**Time to Restore Service**_ : time to restore service when a service incident or a user impacting defect is detected
* _**Change Failure Rate**_ : percentage of releases that result in degraded user experience that requires remediation

These measurements work together to provide four legs holding up a stable platform that is your application.  Trying to optimize one metric at the expense of the others will have negative results.

_Lead Time for Change_ seems to be the one most organizations want to improve the most.  It sounds good on paper, if we get more changes in, we are more responsive to change and it makes the application better.  The problem arises when you are pushing for faster changes, shortcuts are taken.  Tests are not written, code written is not extensible, and more technical debt is accrued.  On some occasions, the new change can cause an outage due to a missed edge condition.  

_Change Failure Rate_ and _Time to Restore Service_ tempers _Lead Time for Change_.  By keeping the _Change Failure Rate_ low or constant as you reduce your _Lead Time For Change_, you can maintain a good quality of code and enjoy a dynamic application.  A good _Time to Restore Service_ measurement provides a good safety net to make the changes.

_Deployment Frequency_ is another metric that seems easy to improve.  There are always changes in the pipeline, just push them out as they are done and the frequency increases.  Again in our push for a better number, substandard code can be deployed as checks and balances are compromised for a single, better score.  

Having a good _Time to Restore Service_ number can mitigate having the site down for a while, but the user experience is still degraded through unplanned down time and lack of new features.  Additionally, these rollbacks will increase the _Change Failure Rate_.  A judicious increase in _Deployment Frequency_ can compliment a reduction in _Lead Time for Change_ as now there is a shorter window to introduce changes.

_Time to Restore Service_ seems like it could stand on its own.  After all, a robust infrastructure and good roll back strategy can mitigate almost anything that can disrupt service.  Highly available, redundant clusters can allow an application to survive the destruction of the primary host datacenter.  Operations could create that dream infrastructure, put in strong change controls and procedures, and the _Time to Restore Service_ number is now sub-second.  

This is a DevOps article, and Development and Operations work together.  Overly burdensome change controls can be hostile to development efforts and negatively affect _Lead Time For Change_ and _Deployment Frequency_.  We have already discussed how this measurement can rein in overzealous increases in _Deployment Frequency_ and decreases in _Lead Time to Change_.

Finally, a good _Change Failure Rate_ is a strong indicator of high quality code.  While not as attractive to the bottom line as the other indicators, it is something Development could drive on their own.  Introduce some automated testing, static code analysis, code linting, documentation and you have some great code, right?  These are all good, but a single minded approach to test coverage can lead to regression test runs that last hours.  Couple this with a requirement to complete a regression for every merge and the _Lead Time For Change_ number is hurt, which also affects the _Deployment Frequency_.

When measuring the performance of your DevOps process, it is easy to be overwhelmed by all the possible measurements.  The four measurements reviewed here are a great place to start or even finish, but for best results, it is important to use all these measurements together.

<a href="https://dev.to/leading-edje">
  ![Smart EDJE Image](https://dev-to-uploads.s3.amazonaws.com/i/5uo60qforg9yqdpgzncq.png)
<a/>