How to Keep Your Master Branch Green with Git
Published: 2020-11-03 12:40:55

---
title: How to Keep Your Master Branch Green with Git
date: 2020-08-15 21:58:00 UTC
published: true
tags: 
canonical_url: https://kevincox.ca/2020/08/15/git-green-ci/
---

<main><p>Most often I see a CI setup that leads to relatively common failed builds on the <code>master</code> branch. The setup I see the most often is:</p>

<ol>
  <li>Prepare a patch/change/PR/MR.</li>
  <li>Run tests on that change. (And get reviews)</li>
  <li>Apply the change to <code>master</code>.</li>
  <li>Run the tests again on <code>master</code>.</li>
</ol>

<p>This probably got adopted as the defacto standard because it is very easy to implement. Just run CI on every commit to the repo. However it makes it fairly easy for logical “merge conflicts” to get into <code>master</code>, which then causes the tests to fail in step 4, and everyone who pulls that branch will see unrelated issues when they run tests on their changes.</p>

<h2 id="latest-green"><a href="https://kevincox.ca/2020/08/15/git-green-ci/#latest-green">Latest-Green</a></h2>

<p>The solution is quite simple, run the tests <strong>on the merged result</strong> before applying the changes to master. A simple solution replaces steps 3 and 4 with:</p>

<ol>
  <li>Apply the changes to <code>staging</code>.</li>
  <li>When <code>staging</code> is green, fast-forward <code>master</code> to <code>staging</code>.</li>
</ol>

<p>This is already a big improvement, <code>master</code> is now the “latest green”, and developers will not be blocked by broken code. However once <code>staging</code> goes red <code>master</code> will stop being updated until someone fixes it. This means that it still delays changes being published, both to production and to other developers for an extended period of time.</p>

<h2 id="auto-revert"><a href="https://kevincox.ca/2020/08/15/git-green-ci/#auto-revert">Auto-Revert</a></h2>

<p>To make this strategy really shine you can apply auto-rollback to the <code>staging</code> branch. The exact policy here can be complicated, however here is a simple policy to start from.</p>

<blockquote>
  <p>If CI fails in <code>staging</code> remove that patch, and re-queue all the patches behind of it.</p>
</blockquote>

<p>This assumes that the commit that failed was the problem, and optimistically assumes that the other commits are good.</p>

<h2 id="downsides"><a href="https://kevincox.ca/2020/08/15/git-green-ci/#downsides">Downsides</a></h2>

<p>The major downside is that it can take a lot of CI resources or add a lot of latency to merges if commits are failing often. This is because you basically have two approaches to running CI when multiple patches are queued.</p>

<ul>
  <li>Optimistic: Merge all of the patches into <code>staging</code> and start CI immediately.</li>
  <li>Pessimistic: Don’t start CI until the previous patch has passed.</li>
</ul>

<p>If your CI is fast either approach will work. However if your time between patches is shorter on average than your CI latency your queue will build using the pessimistic strategy. The solution is to use the optimistic strategy which is optimal if you have no failures. However it will waste a lot of CI resources if a patch fails because all of the jobs behind it will (likely) fail as well, then be retried with a different base.</p>

<p>More complex strategies can be employed as well such as limiting the number of queued patches that CI runs on, if you base this number based on your expected failure rate it can be very effective.</p>

<h2 id="support"><a href="https://kevincox.ca/2020/08/15/git-green-ci/#support">Support</a></h2>

<ul>
  <li><a href="https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.html">GitLab non-free editions support this</a></li>
  <li>
<a href="https://bors.tech/">Bors</a> supports this on GitHub.</li>
</ul>