tweaking the gitlab kubernetes installer
Tags: gitlab, helm, kubernetes, tiller
Published: 2019-10-09 15:18:42

<!-- wp:paragraph -->
<p>GitLab has a lovely feature that can bring up a Kubernetes cluster and install build runners on it with a single click. But you can't tweak it.</p>
<!-- /wp:paragraph -->

<!-- wp:more -->
<!--more-->
<!-- /wp:more -->

<!-- wp:paragraph {"align":"left"} -->
<p style="text-align:left">While that's nice for those who just want to get builds running quickly on their own infrastructure, we were looking for something a little more customizable.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Out of the box</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Although GitLab Cloud can currently only provision a cluster on Google Kubernetes Engine, it can connect to and manage any existing Kubernetes cluster as long as it's reachable via the web (an on-prem GitLab deployment doesn't have this requirement, of course). Once it's authenticated, you can use the GitLab UI to install Helm, which is required for installing the rest of the available applications like Prometheus or GitLab Runner.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I followed the <a href="https://gitlab.com/help/user/project/clusters/index.md#add-existing-kubernetes-cluster">GitLab docs</a> on how to add my cluster, and everything seemed to be on track.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The snazzy UI makes use of the <a href="https://gitlab.com/gitlab-org/charts/gitlab-runner/tree/master">GitLab Runner Helm chart</a> to deploy build runners to your cluster. Although the chart itself offers a lot of customization,  tweaking the chart values isn't an option if you're deploying via the UI. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Schedule conflict</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Our cluster is composed of several node pools of non-uniform size. Specifically, we were looking for a way to tell GitLab to run builds with a certain tag only on the beefier nodes in our cluster. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Unfortunately, at the time of writing, there is no way to define resource requests at the job level. There's an <a href="https://gitlab.com/gitlab-org/gitlab-runner/issues/2902">open issue</a> and even a <a href="https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/874">merge request</a> with some traction addressing this very feature, but no word from the maintainers. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Since the runner chart <a href="https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/master/values.yaml#L281">supports</a> setting a <code>NodeSelector</code> for the runners, what we can do instead is use Kubernetes labels to schedule our runners on a subset of nodes with a particular label, and then specify a tag in our CI job so it only runs on runners with the tag associated with that label.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Our beefy nodes are assigned the label key-value pair <code>runner: large</code>. So we match this in the runner configuration by passing a parameter to Helm:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>--set runners.nodeSelector.runner="large"</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>We also want to make sure that we don't schedule more than one runner per node, so we set a CPU request that's too big for our small nodes (0.94 CPU allocatable), but still schedulable on our big nodes (3.92 CPU allocatable):</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>--set runners.builds.cpuRequests="3"</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Second, we need to assign a tag to our souped-up runners indicating their beefy spec to jobs. This tag, <code>runner-large</code>, will later be specified in our pipeline job:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>--set runners.tags='runner-large'</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Now that we know exactly how we want to configure our runner chart, we have two options: a) deploy Helm, Tiller, and the runner chart on our own, or b) use the UI to deploy Tiller and GitLab Runner to our cluster, then tweak it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There are already a ton of examples out there on how to deploy charts with Helm.<br>I'm going to take you down the back route where we tweak an existing GitLab runner setup provisioned with the UI, without having to re-deploy.</p>
<!-- /wp:paragraph -->

<!-- wp:separator {"className":"is-style-dots"} -->
<hr class="wp-block-separator is-style-dots"/>
<!-- /wp:separator -->

<!-- wp:heading -->
<h2>Lockdown</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you installed Tiller via the GitLab UI, logging in to your cluster and doing a <code>helm ls</code> to see what's what like I did will quickly reveal that something is amiss:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Error: could not find tiller</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>The first non-standard thing GitLab does when it installs Tiller in your cluster is deploying it to its own namespace - <code>gitlab-managed-apps</code> - instead of the default <code>kube-system</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We can tell Helm where to find Tiller either with the command-like flag <code>--tiller-namespace</code> or by setting the environment variable <code>TILLER_NAMESPACE</code>:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$ export TILLER_NAMESPACE=gitlab-managed-apps
$ helm ls --tiller-connection-timeout 30
Error: context deadline exceeded</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Since Helm's default timeout is an obnoxiously long 5 minutes, we shorten it to 30 seconds, only to find out that something is still preventing us from gaining access.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The second non-standard thing GitLab does is it <a href="https://github.com/helm/helm/blob/master/docs/tiller_ssl.md">deploys Tiller with TLS enabled</a>. This makes communication between the Helm client and Tiller secure. But now not only do we need to authenticate to our cluster, we need a <em>separate</em> set of credentials to authenticate to Tiller. Whew.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here's a little shell snippet that will pull the certs and keys out of the Tiller secret and set some environment variables so Helm can use them:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>export HELM_HOME=${HELM_HOME:-$HOME/.helm}
export TILLER_NAMESPACE=gitlab-managed-apps
export HELM_TLS_ENABLE=true
kubectl config set-context --current --namespace gitlab-managed-apps
kubectl get secret tiller-secret -ojsonpath='{.data.ca\.crt}'  | base64 -d > ${HELM_HOME}/ca.pem
kubectl get secret tiller-secret -ojsonpath='{.data.tls\.crt}' | base64 -d > ${HELM_HOME}/cert.pem
kubectl get secret tiller-secret -ojsonpath='{.data.tls\.key}' | base64 -d > ${HELM_HOME}/key.pem</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><strong>Warning!</strong> If you already have Helm TLS certificates in <code>~/.helm</code>, <em>they will be overwritten</em>. <br>Take a look at <a href="https://medium.com/nuvo-group-tech/configure-helm-tls-communication-with-multiple-kubernetes-clusters-5e58674352e2">this post</a> for a guide on juggling multiple Kubernetes clusters with different sets of credentials for Helm. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If all goes smoothly, you should be able to see your runner deployment with <code>helm ls</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Tweak it</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now that we finally have access to GitLab's Tiller, we can follow the official GitLab <a href="https://docs.gitlab.com/runner/install/kubernetes.html#updating-gitlab-runner-using-the-helm-chart">docs</a> on how to update a runner deployed with Helm, adding in our extra flags from above:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>// add GitLab Helm chart repo to our local Helm config
$ helm repo add gitlab https://charts.gitlab.io &amp;&amp; helm repo update
// save current runner chart values
$ helm get values runner > values.yaml 
// merge in our updated configuration
$ helm upgrade --namespace gitlab-managed-apps \
    -f values.yaml \
    --set runners.nodeSelector.runner="large" \
    --set runners.builds.cpuRequests="3" \
    --set runners.tags="runner-large" \
    runner gitlab/gitlab-runner</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Finally, we can confirm that the changes took hold by waiting a moment for the pods to restart, then looking up the manager pod and printing out its config:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$ RUNNER_POD=$(kubectl get pods -l app=runner-gitlab-runner -ojsonpath='{.items[0].metadata.name}')
$ kubectl exec $RUNNER_POD cat /home/gitlab-runner/.gitlab-runner/config.toml</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>With any luck you should see your configuration updated to reflect the new values you passed to Helm when updating your deployment.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now in your <code>.gitlab-ci.yml</code>, you can add </p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">beefy job:
  tags:
    - runner-large</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>for a particular job, and your build will only run on nodes with the <code>runner: large</code> label.</p>
<!-- /wp:paragraph -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:paragraph -->
<p>The moral of this story: If you want custom runner configuration, use your own damn Tiller.</p>
<!-- /wp:paragraph -->

*this post was originally published on [bad gateway](https://badgateway.qc.to),
a collection of notes about dev and ops things.*