⁉ Why I started developing 💡 my new software project  by building a 🚀 Continuous Deployment 🔃 pipeline 
Tags: devops, sre, architecture, pipeline
Published: 2020-03-01 15:41:15

Somehow, in recent years, when discussing software delivery pipelines in modern companies, people would casually drop the term **CI / CD** (which means *Continuous Integration* and *Continuous Delivery / Deployment*) always combining both practices. In many cases it is implied that the two practices always go together, or, maybe *Continuous Delivery* follows *Continuous Integration* as an optional step.

During recent years I had an opportunity to oversee the establishment of a software delivery pipeline twice, implementing only *Continuous Integration* first time, and starting with *Continuous Delivery* in my latest project. 

## 🙈 TL;DR / Conclusion 

There are **HUGE** differences between the two practices. Differences that don't just have impact on the technology used, but also on the culture of the engineering team. Implementing *Continuous Integration* only and hoping that it is a sufficient foundation to switch to *Continuous Delivery* in the future is a myth. 

## 👊 Which is which and why do you need them?

To explain the choices that we made, one needs to understand the difference between **CI** and **CD**. For those who feel confident that they understand it, skipping the below paragraph is recommended. For those who are suddenly not so sure - read on...

### ☝ Continuous Integration

Continuous Integration is a practice targeted at maintaining a working "integrated" system consisting of multiple components being worked on in parallel by multiple engineers / engineering teams.

The following diagram demonstrates a *simplified* Continuous Integration process:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ofs62oem62m3sosz4ahy.png)

Each of the developers involved is delivering a change in one of the components. The CI constructs an environment consisting of the latest components with the new addition and performs a cycle of automated tests.

The need to implement a Continuous Integration in businesses that depend on their ability to deliver software efficiently and reliably is very obvious. The alternative is straight out **painful**. Doing periodic "big bang" integrations, bringing together development projects that were run separately and hoping that they would magically run together - it's a huge challenge. 

I remember the old days where building a multi-component software server in a large R&D organization would result in a very painful iterative integration time period, during which tons of builds would be delivered to QA and would be pronounced "dead on arrival". 

Quite frankly, I cannot think of any good reason why anyone in this day an age would prefer **not** to use *Continuous Integration*. Sure, building it effectively would require investing in *automated testing*, which is yet another thing that just should be done without any reservations. 

The only reasonable situation in which *CI* shouldn't be used, is the case where the organization maintains a significant amount of legacy software components that cannot be validated with automated testing. If this is the case, then, unfortunately, pretty much everything in this article won't apply. Trying to do Continuous Integration without sufficiently reliable automated testing for components is going to generate something of a very little practical value.

### ✌ Continuous Delivery / Deployment

Continuous **Delivery** and Continuous **Deployment** are terms that quite often get used interchangeably. There is a difference between the *canonical* definitions of the two which is important to note, in order to user the correct term:

* [**Continuous Delivery**](https://en.wikipedia.org/wiki/Continuous_delivery) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and, when releasing the software, doing so manually. It aims at building, testing, and releasing software with greater speed and frequency. 
* [**Continuous Deployment**](https://en.wikipedia.org/wiki/Continuous_deployment) is an approach, where the software, after undergoing a set of automated reliability gates, actually gets deployed in the production environment in a fully automated (and controlled) manner.

While Continuous **Delivery** may, indeed, be a natural extension to a well-implemented Continuous **Integration**, **Continuous Deployment** requires a significant amount of investment, both technological and methodological, in order to implement successfully. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xk6c4awxgvpvjcno7xf8.png)

The above diagram depicts the full flow of a *Continuous Delivery* cycle implemented with *Canary [Progressive Delivery](https://thenewstack.io/the-rise-of-progressive-delivery-for-systems-resilience/)* (one of the possible strategies for a responsible automated update of software components in production environments).

## 👟 Implementing **just CI**

In order to implement *Continuous Integration* successfully, an organization would need to invest in a number of technical infrastructural components, as well as to encourage (or even enforce) certain rules on the "Definition of Done" for Software Developers.

Three main components listed below allowed us to enable efficient *CI* and gain a consensus (among the members of our engineering team) that it is a valuable asset for the organization.

### 🤖 Automated Tests

An absolute must for any **CI** environment, automated testing of your code provides the foundation for an ability to *claim* that a code change has been successfully integrated.

A lot has been said about automated testing methodologies. At this point, it is enough to state that testing automation should be introduced on multiple levels (and not just either *Unit Testing* or *End-to-End Testing*). Layers, such as *Module Testing*, *Contract Testing*, *Performance Testing* and even *Automated Security Tests* should be considered, depending on the requirements of your project.

It was our experience, that introducing all relevant tests incrementally for every code change and treating these as an inseparable part of the development cycle (rather than separate tasks) ensured that our ability to successfully gate software changes in a dynamic environment keeps the overall system stable.

### 🌀 CI Pipeline

During the early days of *Continuous Integration*, when it was still called *Software Configuration Management Automation*, organizations used to automate software building, testing and packaging tasks writing custom scripts / code. [Jenkins](https://jenkins.io) was, probably, the first **CI** and later also **CD** pipeline tool that has emerged as the main-stream-adopted (almost de-facto standard) for building *Automated CI Pipelines*.

Additional products, both self-hosted open source ones, as well as SaaS-delivered ([Travis CI](https://travis-ci.com/), [Circle CI](https://https://circleci.com/), [Gitlab Pipelines](https://docs.gitlab.com/ee/ci/pipelines.html), [GitHub Actions](https://https://github.com/features/actions), [Google Cloud Build](https://cloud.google.com/cloud-build) and more) have since taken the architecture and the instrumentation of *CI* lightyears ahead.

### 📦 Artifacts Repository

A very important component in the pipeline, repository for storing the tested / verified "deliverables". *Artifacts Repositories* have evolved from a file-system with a predefined structure to a more sophisticated software solutions, capable of delivering the "artifacts" using various Software Package Management / Distribution systems. 


## 👞 Implementing **CD** on top of **CI**

Do we need to implement *Continuous Integration* first, and then consider working on *Continuous Delivery*? A [great article](https://jhall.io/posts/cd-without-ci/) on the subject suggests, that, while this is a conventional approach, it can, actually be done in a different order (with some unorthodox thinking). While this is a valid opinion, **all** of the *Continuous Delivery* environments familiar to the author of this article are, actually, built on top of *Continuous Integration* pipelines.

On top of the components listed above, following ones will need to be added in order to implement **Continuous Deployment**. Above all, the understanding that every delivery will impact real users of your service is something that needs to be generally acceptable by your engineering team. Engineers should feel confident in delivering changes, knowing that the deployment is done in a responsible manner and that the system can be *observed* properly to make sure that it functions according to expectations. 

### 👏 Operator / Trigger for Deployments

This is, in fact, the only *100% mandatory* component for implementing *Continuous Deployment*. This is what ensures its continuousness. The goal of this component is to trigger deployment process automatically upon either a completion of a *Continuous Integration* for a certain change, or upon any other automated trigger.

There are multiple ways to implement such a trigger, both including custom-developed code and open-source projects, such as [Flux CD](https://fluxcd.io/) or [Spinnaker](https://www.spinnaker.io/).

### 🙌 Progressive Delivery Controller

*Progressive Delivery* is a methodology that ensures gradual adoption of newly delivered components by some users of the overall system (but not by everyone at once). Coupled with *Health Metrics Monitoring*, it can be used for a fully automated responsible roll-out.

Depending on the nature of your service, using *Progressive Delivery* is not a must. If you can replace versions of various components of your software in the production environments without worrying about its impact on the end-users, you don't need to do anything special.

If causing interruptions to current users of your service is a concern, then some sort of a progressive delivery controller is required. Open Source projects, such as [Argo Rollouts](https://github.com/argoproj/argo-rollouts) and [Weaveworks Flagger](https://github.com/weaveworks/flagger) deliver this functionality for Kubernetes environments.

There are various methods for performing *Progressive Delivery*, such as, but not limited to *Blue-Green* or *Canary* upgrade.

### 👈👉 Traffic Split Manager / Load Balancer

Since at various stages of a *Progressive Delivery* process both the "Stable" and the "New" versions of the same component will exist simultaneously, something needs to be able to balance traffic between these two. Most *Progressive Delivery Controllers* are capable of controlling the traffic split management by interacting with notions, such as Load Balancers, Service Mesh controllers, ingress controllers and other components capable of directing North-South, as well as East-West traffic to various instances of destinations.

### 👍👎 Health Metrics Monitoring

The ability to establish whether your component is "healthy", or whether it is functioning according to expectations, is a mandatory part being able to do automated deployment. Without this ability, your developers are "blind", not knowing whether the system is functioning or not.

The monitoring can be done either using *passive* methodology (i.e., your component emits various metrics or logs and something monitors whether their data is reflecting a properly functioning system) or using *active* methodology (periodic probes/sensors are executed against the system, verifying that its response is according to what is expected).

When considering systems for collecting metrics, [Prometheus](https://prometheus.io/) rules supremely as the most widely used open source tool, easily integrated with all the leading *Progressive Delivery Controllers*.


## 💫 Tying it all together

Having established what *Continuous Integration* and *Continuous Deployment* are, you can now judge the requirements for your specific case. The article above has focused on the technical components that are required to achieve either of these capabilities, mentioning that the engineering teams and their culture are also an important factor.

Deciding on the proper strategy for your software service is up to you. One important piece of advice is not to underestimate the challenge of changing the direction in the middle and its impact not just on the technical debt, but on the engineering team.

Choose wisely...

P.S., if you are interested in how we implemented an end-to-end Canary Continuous Deployment, refer to [this excellent blog](https://dev.to/orelimelech_/kubernetes-native-cd-with-fluxcd-flagger-and-linkerd-o69) by [Or Elimelech](https://dev.to/orelimelech_).