DEV3L on DevOps Handbook
Tags: devops, agile, craftsmanship
Published: 2021-12-31 21:38:27

---
title: DEV3L on DevOps Handbook
published: true
description: The DevOps Handbook is a guided tour through the world of DevOps. The book provides principles, proven practices, and case studies for organizations that want to build safe and resilient systems, with minimal maintenance cost, and unimaginable levels of business agility.
tags: devops, agile, craftsmanship
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x7dwr5abuoq0echx7hgp.jpg
---

DevOps is good for business - this is undeniable. Elite performers consistently outperform their peers in the market - regardless of the market segment.

**The DevOps Handbook, Second Edition - How to Create World-Class Agility, Reliability, & Security in Technology Organizations**, by *[Gene Kim](https://twitter.com/RealGeneKim)* and *[Jez Humble](https://twitter.com/jezhumble)*, is a guided tour through the world of DevOps. The book provides principles, proven practices, and case studies for organizations that want to build safe and resilient systems, with minimal maintenance cost, and unimaginable levels of business agility.

---

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ez9f2lrudpd2c75ifu6n.png)

```
DevOps isn't about automation,
just as astronomy isn't about telescopes
```
 
Fast flow of planned work to production

**How technology work is managed and perform as leading indicator of market success**

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jbsb14jd565mpvp0vkge.png)

**Downward Spiral** - traditional goals of operations (stability) and development (change) in opposition
Cannot see the big picture when pickled

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nsfzhfh1oiyietm5zjoi.png)

Creating systems that cause feelings of powerlessness is one of the most damaging things we can do to fellow human beings

Deploy during business hours,
lead times in minutes to hours

*Empirically great for business*
Proven better business outcomes [Accelerate](https://itrevolution.com/accelerate-book/)

Loosely coupled teams scale

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u628fz0uenzhkqhww4wj.png) 


# Part I - The Three Ways

- Flow (capabilities and practices)
- Feedback and monitoring
- Continuous learning (generative culture)

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d5ykt4hdp9hkhs8ghn1t.png)
 
[10+ Deploys Per Day](https://www.youtube.com/watch?v=LdOe18KhtT4) - 2009 *[John Allspaw](https://twitter.com/allspaw)*

### Configuration and Infrastructure managed using craftsmanship principles

**Improvement Kata** - daily practice improves outcomes...

`Avoid cargo cult Lean/Agile`

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/edxe2xmbl1u46tumnp7f.png)

**Value Stream** - sequence of activities necessary to produce value

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kl8f53t2uw4bkjptjm3d.png)
 
**Deployment Lead Time** - time from *Dev Complete* to *Production*

**Shift Quality Left** - build quality in, not tacked on at the end

`Process Time (Touch Time) = Lead Time - Wait Time`

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5gbjwwwfdinis0l440hp.png)
 
[Flow Framework](https://flowframework.org/) by *[Mik Kersten](https://twitter.com/mik_kersten)*

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dlr643n2126r8dve88ny.png)

## The First Way

- increase flow
- make work visible - [Kanban](https://prokanban.org/)
- theory of constraints [Goal](https://www.goodreads.com/book/show/113934.The_Goal)
- reduce batch size
- eliminate waste

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1lqgn62zotkyxoellv4i.png)

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ofwi9h7jowd7xxxrjwwx.png)

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qg2l2tuo9bbg89edfo71.png)

## The Second Way

- fast feedback loops
- establish cause and effect
- enables learning

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v155q8wz6ro9vfh8dohl.png)
 
`Continually automate manual tasks - self service`

**Technical Andon Cord** - defect / second opinion - safety culture

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a897eij26hsd1ml06mh4.png)
 
Push decision making down to where work is

## The Third Way

- continuous learning

Transform local discoveries into global improvements

**Blameless Post Mortem** - learn from failures, not blame

```
More important than daily work,
Improvement of daily work 
```

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ma0t3k5geihwz1ori1mw.png)
 
Resilience through anti-fragility
Experiment iteratively towards True North goal


# Part II - Where to Start

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/479e7g45g3m6hhji2fsz.png)
 
Thought process to guide decisions, actual steps to be taken, and case studies to visualize

Start with **enthusiastic early adopters** - find early winds - land and expand

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vcmkk0cc3wljhp3fqtzb.png)
 
`Little fish learn to be big fish in little ponds`

Focus on wait times and rework
Invest 20% (at least) in the "ilities" (NFRs)
Shared pain reinforces shared goals

Design systems with **Conway's Law** in mind

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wgwmad4b6mwdlhy9jnvt.png)
 
# Part III - Technical Practices of Flow

Environments like cattle, not snowflakes
Production like on demand

`Design systems/architecture for testability`

**Immutable Infrastructure** - all artifacts, source, and configuration in version control - removes variance 

*"Done"* includes potentially shippable code deployed in a production-like environment

**Automated testing is principle/foundational** - manual testing cannot scale

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lgoqdcp1ma8g9yv838r2.png)
 
**CI+** - Continuous Integration built upon DevOps practices

Slow and periodic feedback kills development effectiveness
... especially at scale

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tc0wbyujezldcfau7k3z.png)

```
Non-idempotent test should be rewritten, or removed
```

### Optimize for team productivity
- Continuous Integration
- Trunk Based Development

*Deploy to production as frequently as possible*
Every commit for single piece flow
Boring deployments lead to high agility

### Decouple deployment and release
- blue/green (environment)
- feature flags (code)

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/37e3xgplwqam3ydmhni5.png)
 
```
Continuous Integration <
Continuous Delivery    <
Continuous Deployment
```

[Resource Utilization Trap](https://youtu.be/CostXs2p6r0)
**Flow Efficiency** over
*Resource Efficiency*

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/civ7qiusqrb4xr0zblr7.png)

Don't Rewrite,
Delegate and Decommission
**Strangler Fig Pattern**

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dn7ky6kexfp691la8z3l.png)


# Part IV - The Second Way - Technical practices of feedback

Logging, monitoring, and visualizing telemetry/events as a first principle

`Make tracking anything easy`

**Application Performance Monitoring** - if it's important enough to implement, it's important enough to instrument

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9v6vbcgq8fgqwx2rlx88.png)
 
Allow self service of metrics

Overlay system events - such as deployments, incidents, and/or maintenance periods - on top of business metrics

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fzqj3xmmr4fkpntifw5v.png)

Use means and standard deviations to detect anomalies in clusters - do not need to know what normal is, just what it is not

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ijfqzmbudue6e7k3bty.png)

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cr9v3rswemkl09d5rp44.png)

**Rotate Pager Duty** through the entire team - empathy and visualization - Goldilocks alerting (just right)

Validate assumptions empirically in lowest fidelity possible


# Part V - The Third Way - The Technical Practices of Continual Learning and Experimentation

`Pair programming is live peer review`

- one repository to rule them all
- codify non-functional requirements
- create run books for manual operations
- guidance over Governance on architectural decisions

**In complex systems errors are inevitable**, utilize resilience engineering practices to accommodate - enable chat rooms to trigger events, notify status/results, and alerts

Perform **Blameless Postmortems** after production incidents (and near misses) - lower failure signals as error detection decreases

Retrospect Early, Retrospect Often
Make results globally available

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l96nh47cegyfjf879eb2.png)

Transform documented knowledge, processes, and design standards into reusable code

Create regular space for improvement blitzes - explicit time dedicated to learning and/or improvement of daily work - ensure cross pollination of teams/business units

Allow everyone to teach and learn - most valuable thing employee can do is teach someone  or learn something new

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zt4ysxwudbwjltfvr0tp.png)

Remove "I do not have time to test" as an excuse - [Make It Easy](https://www.youtube.com/watch?v=Jtt7PAejrFA)

```
Make the change easy,
Then make the easy change
```

Automated testing, and automation in general, is foundational to all other DevOps practices

# Part VI - The Technical Practices of Integrating Information Security,  Change Management, and Compliance

Integrate information security practices into daily work - shift left - *compliance by demonstration*

Spread **[OWASP](https://owasp.org/Top10/)** Top 10 throughout organization - create security learning opportunities

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xf67guohl5piev64zcet.png)

Utilize static/dynamic security analysis inside delivery pipeline - source, dependencies, and sub-dependencies are scanned for vulnerabilities

Create paved road for developers to follow - all necessary information security checks - internal package management

Classify *Change Types* by risk
- routine
- standard
- urgent

**Change Management Traceability** - link deployments to commits, and commits to work tickets

Physically and logically separate components that require special compliance - do not force strict policies where not needed

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l1elfacqh8jt71scqik5.png) 

---

Reference Journal Events: [Start](https://twitter.com/dev3l_/status/1476242693311410176), [Finish](https://twitter.com/dev3l_/status/1476779918335881223)