Setting up a "confidential" GitLab🦊🔒
Tags: tutorial, devops, opensource, tooling
Published: 2022-11-20 11:50:06

Hey there. If you ever programmed something "sensitive" or, like me, are managing private meeting notes with Git, you probably know this uneasy feeling when pushing to GitHub. It's simply impossible to tell who may get access to your private repos. There could be malicious insiders leaking stuff or there could be bugs allowing other users to access data etc. 

![GitLab cloud attack surface](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hn045l52iyv7s4kck4uo.png)

This uneasy feeling is one of the primary reasons for people to use self-managed [GitLab](https://about.gitlab.com/). Especially companies concerned about leakage of IP like to run GitLab on-prem. However, maintaining on-prem infrastructure is expensive, tedious, and often not even an option. 

## Confidential computing to the rescue!
Recent server CPUs from AMD and Intel come with so called *confidential computing* features. With these, sensitive workloads can be kept encrypted during runtime in memory and are shielded from the rest of a system. Not even system administrators, cloud provider employees or privileged attackers can access workloads protected this way. And this property can even be verified remotely 🤯

Confidential computing effectively lets you create your own runtime-encrypted and isolated "on-prem" on the public cloud. It's like having your cake and eating it 🍰

If you want to learn more about confidential computing, I recently published a [whitepaper](https://content.edgeless.systems/hubfs/Confidential%20Computing%20Whitepaper.pdf) on it. The [Confidential Computing Consortium](https://confidentialcomputing.io/) also has some good resources. 

## How to make GitLab🦊 "confidential"?
Ok, now that we sort of know what confidential computing is - how do we apply it (correctly) to GitLab and the software it relies on, e.g., PostgreSQL, NGINX, and Redis?

Easy! I recently [posted](https://dev.to/edgelesssys/we-created-an-always-encrypted-kubernetes-1381) about our open-source project [Constellation](https://github.com/edgelesssys/constellation). Constellation is the first *confidential Kubernetes distribution*. Think Rancher Kubernetes Engine (RKE) or RedHat OpenShift for confidential computing. 

Constellation encrypts and shields entire K8s clusters with the help of confidential computing. Basically, Constellation ensures that all data in a cluster is always encrypted (at runtime, at rest, and in transit) and that the cluster cannot be accessed from the infrastructure. 

So making GitLab "confidential" is as simple as running it with Constellation on confidential computing enabled hardware, which is available in Azure and GCP.

## Running GitLab🦊 on Constellation💫
The first step is to [download and install](https://docs.edgeless.systems/constellation/getting-started/install) the Constellation CLI.

Next, a Constellation cluster can be [created in a few steps]
(https://docs.edgeless.systems/constellation/getting-started/first-steps). In essence:
```bash
constellation config generate azure
constellation create --control-plane-nodes 1 --worker-nodes 2 -y
constellation init
export KUBECONFIG="$PWD/constellation-admin.conf"
```

We can now connect to the cluster with `kubectl` or other tools using the auto-generated `constellation-admin.conf`. The config ensures that the connection "confidential" and terminates inside the correct cluster. (No man-in-the-middle possible.)

### Installing GitLab (manually)
We want our "confidential" GitLab to be accessible from the Internet under its own URL. For this, we need to install and configure [external-dns](https://artifacthub.io/packages/helm/bitnami/external-dns) in our freshly created cluster.

Further, we need to install Constellation's [CSI driver](https://docs.edgeless.systems/constellation/workflows/storage) for Azure. The driver ensures that all data written to cloud storage from the cluster gets encrypted automatically. With the upcoming v2.3 release of Constellation, it will be installed by default.

Finally, we can install GitLab via [the official Helm chart](https://docs.gitlab.com/charts/installation/), which also installs Redis, PostgreSQL, NGINX, and other components in the cluster. 

### Installing GitLab (automated)
We also created a [custom Helm chart](https://github.com/edgelesssys/constellation-gitlab) that installs and configures all of the above. This can be used as follows:

```bash
export GODADDY_API_KEY=<your creds here>
export GODADDY_SECRET_KEY=<your creds here>
export TARGET_DOMAIN=<your domain, e.g. gitlab.edgeless.systems>
export TLS_ISSUER_EMAIL=<your e-mail address>
helm dependency update ./gitlab
helm upgrade gitlab-infra ./gitlab --install --namespace default --set infra.enabled=true --set apiKey=$GODADDY_API_KEY --set secretKey=$GODADDY_SECRET_KEY
helm upgrade gitlab-app ./gitlab --install --timeout 600s --set app.enabled=true --set gitlab.global.hosts.domain=$TARGET_DOMAIN --set gitlab.certmanager-issuer.email=$TLS_ISSUER_EMAIL --namespace gitlab --create-namespace
```
Note that the above assumes you're using GoDaddy as your domain provider. It should be easy to adapt to other providers.

## Live demo📺
I deployed our confidential GitLab to https://gitlab.edgeless.systems/ and synced the Constellation repo from GitHub.

## What are the security properties?
Constellation ensures that the entire K8s cluster runs in runtime-encrypted and isolated Confidential VMs (CVMs) and that all data that leaves the CVMs remains encrypted. For example, if Redis or PostgreSQL write data to cloud storage that data is automatically encrypted. The corresponding cryptographic keys are all generated and managed within CVMs. We as DevOps engineers don't need to worry about these. Bottom line is that all data is always encrypted, even during runtime in memory.

Constellation also verifies the integrity and authenticity of all CVMs and that they all are running the same "good" Constellation node image. This way, once `constellation init` completes, you know that you are actually talking to an end-to-end confidential cluster via `kubectl`.

You can learn more about these concepts in the Constellation [docs](https://docs.edgeless.systems/constellation/overview/confidential-kubernetes).

### What about user connections?
As I discussed already in a previous post on [confidential Rocket.Chat](https://dev.to/flxflx/rocketchat-constellation-most-secure-chat-server-ever--50oa), users of the service should also have means to verify that the service is confidential and that it is the right instance they're talking to. 

For browser-based connections, currently, the only way is to manually check the fingerprint of the TLS certificate. In the case of our demo instance of GitLab (https://gitlab.edgeless.systems/) the SHA-256 fingerprint is `CB C2 F9 23 7B 46 66 FC 5B 88 70 D5 3C 73 24 98 B4 EC 1D 12 EC 38 85 8C FA A6 44 B2 43 73 34 18`🧐

For Git CLI-based connections, things are simpler. By adding the following entry to our `gitconfig` file, we can ensure that Git always checks that the right certificate is present when talking to https://gitlab.edgeless.systems/:

```
[http "https://gitlab.edgeless.systems/"]
	pinnedPubkey = sha256//aZTzzCepU+Sa34+xkqyFGiWXG+/yHtF6Q5AgMMAjHhs=
```

With this in place, we know for sure that commands like `git clone https://gitlab.edgeless.systems/edgelesssys/constellation.git` or a corresponding `git push` will always go to the right confidential instance of GitLab and can't be diverted or intercepted by an attacker in control of the host of `gitlab.edgeless.systems` or the corresponding DNS entries.

![Confidential Gitlab user](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/80mu4k0yk674toatpley.png)

## Wrap up

Hope you enjoyed the quick walkthrough. Personally, I have to admit that I love the idea of having a GitLab running on Azure, which can't be accessed from the infrastructure and keeps all of the data always encrypted. 

![I don't always Git](https://i.imgflip.com/71aym7.jpg)

What do you think? Any suggestions for apps we should make "confidential" next?