Kubernetes Private Docker Registry With Distribution Registry
Tags: kubernetes, devops, containers, linux
Published: 2023-09-11 00:56:40

{%- # TOC start (generated with https://github.com/derlin/bitdowntoc) -%}

- [NFS Server](#nfs-server)
  * [Package Installations](#package-installations)
  * [Exports](#exports)
- [Persistent Volume Setup ](#persistent-volume-setup)
- [Certificate Setup](#certificate-setup)
  * [Certificate Signing](#certificate-signing)
  * [Secret Setup](#secret-setup)
- [Load Balancer](#load-balancer)
- [Ingress](#ingress)
- [Registry Basic Setup](#registry-basic-setup)
  * [Ingress Setup](#ingress-setup)
  * [Testing](#testing)
- [Conclusion](#conclusion)

{%- # TOC end -%}

When working with container images you can get them from a registry or use custom build ones. However, custom built ones need to be in a place where kubernetes recognizes them on all nodes. This is easy if you use something like Docker Hub but another solution will be required for custom built internal images. This article will look at a very simple container registery hosted on Kubernetes and persistent storage provided through NFS.

## NFS Server

In order for this container setup to work on Kubernetes, there needs to be a permanent file storage that lives beyond pod lifetimes. This will be achieved through the Network File System (NFS) protocol. Support for this is available in most Linux kernels and the client/server packages are available on most major distributions.

### Package Installations

This setup will require both an NFS server to be installed and NFS client tools available on all Kubernetes nodes. For this task I'll use a Raspberry Pi with a 128GB USB storage drive attached to it. First for the server installation:

```
$ sudo apt-get update
$ sudo apt-get install -y nfs-kernel-server
```

On non-NFS server nodes I'll install `nfs-common` so they're able to interact with NFS:

```
$ sudo apt-get update
$ sudo apt-get install -y nfs-common
```

Next I'll setup a directory on the USB drive specifically for container storage. I'll also ensure it's owned by `nobody:nogroup` which is what NFS will run under:

```
$ sudo mkdir /mnt/usbdrive/containers
$ sudo chown nobody:nogroup /mnt/usbdrive/containers
```

And finally an `/etc/fstab` entry:

```
UUID=d3401f62-06ba-4737-8e19-c35c5a688fdc /mnt/usbdrive ext4 defaults 0 2
```

The UUID I obtained via:

```
# blkid
<snip>
/dev/sda1: UUID="d3401f62-06ba-4737-8e19-c35c5a688fdc" BLOCK_SIZE="4096" TYPE="ext4"
```

This allows for more reliable drive targeting than using the standard `/dev` nodes.

### Exports

Now exports will be needed for exposing directories to consumers of the NFS share. In this case it will be the docker registry container, though it could be used for other purposes as well. Given I'm the only one using the local network I'll allow the local network CIDR access:

```
/mnt/usbdrive/containers/ 192.168.1.0/24(rw,async,no_subtree_check)
```

- `rw`: Just making sure containers can be both read and written
- `async`: I ended up enabling this for performance purposes
- `no_subtree_check`: While I don't see there being much renames, I still chose this as an option to be on the safe side stability wise

I'll save the file and restart the server to enable the export changes:

```
$ sudo systemctl restart nfs-kernel-server
$ sudo systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2023-09-09 00:26:41 BST; 5s ago
    Process: 2318294 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 2318295 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
   Main PID: 2318295 (code=exited, status=0/SUCCESS)
        CPU: 16ms
```

## Persistent Volume Setup 

With NFS setup we can integrate it with Kubernetes as a [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). The overall setup looks like this:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: private-registry
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-registry
  labels:
    k8s-app: private-registry
spec:
  storageClassName: nfs
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  mountOptions:
    - wsize=32768
    - rsize=3276
    - hard
    - rw
    - nolock
    - nointr
  nfs:
    path: /mnt/usbdrive/containers
    server: 192.168.1.93
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-registry
  namespace: private-registry
  labels:
    k8s-app: private-registry
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
```

First a namespace is created for our registry components. Then a PersistentVolume will describe the overall storage properties, including NFS connection. Be sure to replace `<serverIP here>` respectively. There's a number of mount options which are also performance related, as I found docker uploads become really unreliable/hang without them. Values given are mostly taken from a [somewhat related GitHub issue](https://github.com/docker/for-mac/issues/6544). PersistentVolumeClaim takes a portion of the PersistentVolume for use and also ties it to the namespace. In this case it's exclusively going to be used for container purposes so the full capacity is allocated. Now to make a test pod to ensure everything is working properly:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: nfs-test
  namespace: private-registry
spec:
  volumes:
    - name: nfs-test-storage
      persistentVolumeClaim:
        claimName: pvc-registry
  containers:
    - name: nfs-test-container
      image: alpine:latest
      command: ['sh', '-c', 'sleep 99d']
      volumeMounts:
        - mountPath: "/tmp/test"
          name: nfs-test-storage
```

Then apply and test that everything works:

```
$ kubectl apply -f nfs-test.yaml
pod/nfs-test created
$ kubectl exec -i -t -n private-registry nfs-test -- /bin/sh
/ # ls -lah /tmp/test
total 8K
drwxr-xr-x    2 nobody   nobody      4.0K Sep  8 22:11 .
drwxrwxrwt    1 root     root        4.0K Sep  9 00:37 ..
/ # touch /tmp/test/test.txt
```

To ensure NFS is completely working I'll check the USB drive backing the NFS server:

```
$ ls -lah /mnt/usbdrive/containers/
total 8.0K
drwxr-xr-x 2 nobody nogroup 4.0K Sep  9 01:38 .
drwxr-xr-x 4 root   root    4.0K Sep  8 23:11 ..
-rw-r--r-- 1 nobody nogroup    0 Sep  9 01:38 test.txt
```

## Certificate Setup

For this use case I'll have the registry available over HTTPS. As far as the certificates go they can either be self-signed or from a certificate authority. In my case I just [happen to have a certificate authority](https://dev.to/cwprogram/create-your-own-local-root-ca-with-yubikey-signing-4g72) so I'll go ahead and use that to first create the registry certificate. 

### Certificate Signing

First an openssl configuration:

`docker-registry.cnf`
```ini
[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
C = US
ST = Texas
L = Dallas
O = cwprogram
OU = cwprogram Cert Authorization
CN = registry-auth.cwprogram.com

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = registry.cwprogram.com
DNS.2 = registry.private-registry.svc.cluster.local
```

This will provide two separate DNS entries for the cert. The first is for viewing from the outside (resolved via `/etc/hosts`) and the other is for when Kubernetes pods will interact with it. The format of service DNS resolution is:

`[service-name].[namespace].svc.cluster.local`

Now for generation and signing of the cert:

```
$ openssl ecparam -genkey -name prime256v1 -out docker-registry.pem
$ openssl req -config ./docker-registry.cnf -key docker-registry.pem -new -sha256 -out ./intermediateCA/csr/docker-registry.csr
$ openssl ca -config ./openssl_intermediate.cnf -engine pkcs11 -keyform engine -extensions server_cert -extensions req_ext -extfile docker-registry.cnf -days 375 -notext -md sha256 -in ./intermediateCA/csr/docker-registry.csr -out ./docker-registry.crt
```
Given that I'm using an intermediate signing key (this step is not necessary for self signed) the final certificate needs to be a bundle of the server and the intermediate certificate:

```
$ cat docker-registry.crt ./intermediateCA/certs/intermediate.cert.pem > docker-registry.bundle.crt
```

On a final note you'll also need the Root CA cert or self-signed cert:

- Trusted by all nodes
- Trusted by container system trust store if you plan to build docker images within a container

You can check [this page](https://thomas-leister.de/en/how-to-import-ca-root-certificate/) for a nice list of how to trust certs for various operating systems and Linux distributions. 

### Secret Setup

I'll go ahead and setup a TLS secret for the registry:

```
$ kubectl create secret tls docker-registry --cert=docker-registry.bundle.crt --key=docker-registry.pem -n private-registry
```

Kubernetes secrets have a Docker Registry (we'll be using this later) and TLS special storage that make it easy to store special secrets. The TLS one in particular excepts both the cert and key for us making the process easy. Now to verify our secrets:

```
$ kubectl describe secret -n private-registry
Name:         docker-registry
Namespace:    private-registry
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1779 bytes
tls.key:  302 bytes
```

`kubectl` is smart in showing the certificate information so that we know they're present without revealing the underlying secret data. I'll also store my Root CA as well to make it available for configuration later:

```
$ kubectl create secret generic root-cert --from-file=tls.crt=./cwprogram_root.crt -n private-registry
```

I'm not using the TLS secret type here because I'd rather not store my Root CA key in a less than ideal security lacking location. Instead I'll create a generic secret that stores the key `tls.crt` with the contents of my root certificate. I've also bound it to the `private-registry` namespace.

## Load Balancer

One issue to deal with is that we need an IP address to resolve our custom host name to. The main way to handle this would be a LoadBalancer service type. Unfortunately, Kubernetes supported LoadBalancer backends tend to be cloud provider solutions like AWS ELB. Given that my cluster is hosted on bare metal this won't quite work out. To work around this issue I'll be utilizing [MetalLB](https://metallb.universe.tf/). Before beginning though I'll need to meet some prerequisites regarding `kube-proxy`: 

```
$ kubectl edit configmap -n kube-system kube-proxy
```

The main things to change here is ensuring `mode: "ipvs"` and `strictARP: true`. Once that's done save and quit the edit session and then do a rolling update on `kube-proxy`:

```
$ kubectl rollout restart daemonset kube-proxy -n kube-system
```

This will ensure it picks up the new changes. Now it's time for the actual metallb installation:

```
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.11/config/manifests/metallb-native.yaml
```

Immediately after doing this I'm going to get rid of a component called "speaker". This is meant to do some BGP protocol work to assign IPs, but I'm using Calico which handles that for us:

```
$ kubectl delete ds -n metallb-system speaker
```

Now the way that this will work is Calico and MetalLB will share some IP information. The main requirement is that the IP addresses we chose do not overlap with DHCP. Looking at my router DHCP starts at `192.168.1.64` and ends at `192.168.1.253`. Using the CIDR `192.168.1.0/26` will get me to about that IP allocation wise. For this to work I'll need to adjust Calico's settings. This is based off the work by [Edward Cooke's setup article](https://www.frakkingsweet.com/metallb-calico-bgp-loadbalancer-and-loadbalancer-ips/). First I'll get a base YAML to hold the BGPConfig that Calico will use:

`bgp-config.yaml`
```yaml
apiVersion: crd.projectcalico.org/v1
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  asNumber: 65000
  nodeToNodeMeshEnabled: true
  serviceLoadBalancerIPs:
```

Then I'll explicitly declare each IP that's part of my CIDR to enable Source IP tracking later on:

```
for i in $(seq 1 62); do echo "  - cidr: 192.168.1.$i/32"; done >> bgp-config.yaml
```

With that complete I'll finish the setup:

```
$ kubectl apply -f bgp-config.yaml
$ sudo calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 192.168.1.91 | node-to-node mesh | up    | 18:25:58 | Established |
| 192.168.1.93 | node-to-node mesh | up    | 18:25:58 | Established |
+--------------+-------------------+-------+----------+-------------+
```

I've also verified that the BGP routing with Calico isn't extremely broken by this change. Finally I'll setup the IP pool that MetalLB will use to assign to LoadBalancer type services:

`ip-pools.yaml
```yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: service-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.0/26
```

This is a newer custom resource method that MetalLB uses which deprecates the previous config map and extends across the IPs I declared in my BGPConfig. Given that Calico is handling most everything the layout is pretty simple. All that's left is to apply it:

```
$ kubectl apply -f ip-pools.yaml
```

## Ingress

In order to setup the registry I'll be utilizing an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/). However, out of the box there's nothing handling that ingress. An ingress provider needs to be installed for this purpose. I'll go ahead and use [haproxy-ingress](https://haproxy-ingress.github.io/) to meet this ned. The installation requires [Helm](https://helm.sh/), a Kubernetes package manager and templating system. Installation will be done through a dpkg repository:

```
$ curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
$ sudo apt-get install apt-transport-https --yes
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install -y helm
```

I'll add the repository pointing to haproxy-ingress so helm knows where to pull from:

```
$ helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts
```

Making this work with our setup requires a slight adjustment to the configuration, so create the following new file:

`haproxy-ingress-values.yaml`
```yaml
controller:
  hostNetwork: true
  kind: DaemonSet
  ingressClassResource:
    enabled: true
  service:
    externalTrafficPolicy: "Local"
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule
```

The `hostNetwork` is the more essential option here as it ensures the resulting pods will be bound to the IPs of each node. This is useful for accessing the registry from local systems outside of kubernetes nodes. It's also setup as a DaemonSet to be on all nodes, with a toleration to include the control plane in that making it even more accessible from the outside. The "Local" external traffic policy keeps source IP information thanks to the `serviceLoadBalancerIPs` work that was done earlier. Now for the final installation:

```
$ helm install haproxy-ingress haproxy-ingress/haproxy-ingress \
  --create-namespace --namespace ingress-controller \
  --version 0.14.5 \
  -f haproxy-ingress-values.yaml
```

After everything is spun up:

```
$  kubectl --namespace ingress-controller get services haproxy-ingress -o wide
NAME              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
haproxy-ingress   LoadBalancer   10.104.252.216   192.168.1.0   80:30368/TCP,443:31766/TCP   8s    app.kubernetes.io/instance=haproxy-ingress,app.kubernetes.io/name=haproxy-ingress
```

Haproxy ingress is now setup to start acting as an ingress backend for our services.

## Registry Basic Setup

The registry itself is fairly simple and bundled [as part of distribution](https://github.com/distribution/distribution). Note that if you're looking for a more scalable solution something like [Harbor](https://goharbor.io/) might be a good fit. First I'll define a deployment and service for the registry:

`docker-registry.yaml`
```yaml
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: private-registry
spec:
  selector:
    k8s-app: docker-registry
  ports:
  - name: https
    protocol: TCP
    port: 4343
    targetPort: 4343
  selector:
    app: docker-registry
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
  namespace: private-registry
  labels:
    app: docker-registry
spec:
  replicas: 3
  selector:
    matchLabels:
      app: docker-registry
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - image: docker.io/registry:2
        name: docker-registry
        env:
          - name: REGISTRY_HTTP_SECRET
            value: "[redacted]"
          - name: REGISTRY_HTTP_ADDR
            value: "0.0.0.0:4343"
          - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
            value: "/var/lib/registry"
          - name: REGISTRY_HTTP_TLS_CERTIFICATE
            value: "/certs/domain.crt"
          - name: REGISTRY_HTTP_TLS_KEY
            value: "/certs/domain.key"
          - name: REGISTRY_STORAGE_DELETE_ENABLED
            value: "true"
        ports:
        - containerPort: 4343
          protocol: TCP
        volumeMounts:
        - name: registry-mount
          mountPath: "/var/lib/registry"
        - name: docker-certificate
          subPath: tls.crt
          mountPath: "/certs/domain.crt"
        - name: docker-certificate
          subPath: tls.key
          mountPath: "/certs/domain.key"
      volumes:
        - name: registry-mount
          persistentVolumeClaim:
            claimName: pvc-registry
        - name: docker-certificate
          secret:
            secretName: docker-registry
```

First there's a Deployment resource with quite a lot happening. The NFS share is mounted to `/var/lib/registry` which is set as the directory where layer data and other information will go. I've setup `0.0.0.0:4343` as the bind address for the port exposure to work properly. The registry certificate and key are mounted to a path where they can be exposed to the proper environment variables. `REGISTRY_HTTP_SECRET` is simply a random string I set that is primarily for load balancing purposes (which haproxy is) and `REGISTRY_STORAGE_DELETE_ENABLED` ensures I can easily clean up images through the registry service. A service is then connected to it exposing the proper ports and of type `ClusterIP`. I chose this as it makes it easier to interact with inside the pod network. `registry` is chosen as the service name to match the DNS host on the SSL certificates. Now it's time to apply the changes:

```
$ kubectl apply -f docker-registry.yaml
```

Assuming everything is setup properly the pods should show up rather quickly. 

### Ingress Setup

Now all that's left is to create the ingress:

```yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: docker-registry-ingress
  namespace: private-registry
  annotations:
    ingress.kubernetes.io/secure-backends: "true"
    ingress.kubernetes.io/backend-protocol: "HTTPS" 
    haproxy.org/server-ssl: "true"
    haproxy.org/server-ca: "private-registry/root-ca"
spec:
  ingressClassName: haproxy
  rules:
    - host: registry.cwprogram.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: registry
                port:
                  number: 4343
  tls:
  - secretName: docker-registry
    hosts:
    - registry.cwprogram.com
```

First, `ingressClassName` is set to `haproxy` to ensure it's using the haproxy-ingress controllers. This also uses the HTTPS certificate for the registry so external sources see the properly verified one. The root-ca secret that was also made a bit ago is now referenced here. This is because it will be talking to the docker registry backend over an encrypted connection (that's set by backend-protocol and secure-backends) and needs a way to verify the cert. Anything that hits the ingress with the hostname `registry.cwprogram.com` will be directed to the `registry` service. Now it's time to apply:

```
$ kubectl apply -f registry-ingress.yaml
```

Now I'll put an `/etc/hosts` in all node's `/etc/hosts` that points to the `haprox-ingress` service's IP:

```
192.168.1.0     registry.cwprogram.com
```

### Testing

It's time to validate with some testing. I'll pull an alpine image with arm64 (since my workers are Raspberry PIs), tag it, then attempt to upload:

```
$ docker pull --platform linux/arm64 alpine:latest
latest: Pulling from library/alpine
9fda8d8052c6: Pull complete
Digest: sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest
$ docker tag alpine:latest registry.cwprogram.com/alpine:latest
$ docker push registry.cwprogram.com/alpine:latest
The push refers to repository [registry.cwprogram.com/alpine]
b2191e2be29d: Pushed
latest: digest: sha256:b312e4b0e2c665d634602411fcb7c2699ba748c36f59324457bc17de485f36f6 size: 528
```

I can also interact with the web API:

```
$ curl https://registry.cwprogram.com/v2/_catalog
{"repositories":["alpine","gitea/act_runner","jenkins-python-agent","ubuntu"]}
```

Now to make sure the actual images are working:

`pullpod-test.yaml`
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: pulltest
  namespace: private-registry
spec:
  containers:
    - name: pulltest
      image: registry.cwprogram.com/alpine:latest
      command: ['sh', '-c', 'sleep 99d']
```

A simple test run:

```
$ kubectl apply -f pullpod-test.yaml
pod/pulltest created
$ kubectl exec -n private-registry -i -t pulltest -- /bin/sh
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/aarch64/APKINDEX.tar.gz
v3.18.3-154-g9fca9473248 [https://dl-cdn.alpinelinux.org/alpine/v3.18/main]
v3.18.3-160-gd72f0485282 [https://dl-cdn.alpinelinux.org/alpine/v3.18/community]
OK: 19939 distinct packages available
/ #
```

So at this point you have a perfectly usable registry if you don't care about authentication. You can also see proper source IPs thanks to the `serviceLoadBalancerIPs` configuration we did:

```
192.168.1.60 - - [10/Sep/2023:23:33:46 +0000] "PUT /v2/alpine/manifests/latest HTTP/1.1" 201 0 "" "docker/24.0.5 go/go1.20.3 git-commit/24.0.5-0ubuntu1~22.04.1 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.5 \\(linux\\))"
192.168.1.91 - - [10/Sep/2023:23:36:27 +0000] "HEAD /v2/ubuntu/manifests/latest HTTP/1.1" 200 529 "" "containerd/v1.7.3"
```

## Conclusion

This includes a look at getting a private registry for Kubernetes setup using distribution's registry. Originally I thought of including authentication with this, but given that this is geared towards a local LAN setup authentication is not that much of a concern for most I'd imagine. I'll be looking at authentication in the next installment of this series instead. The fun I had with this including:

- Lots of hours spent trying to get MetalLB working just to realize it was because of using Calico for my Kubernetes networking was causing BGP conflicts
- Most of the complexity is because LoadBalancer services are dead simple if you're using some kind of managed service like GKE, forcing you into a BGP nightmare if you're doing bare metal
- Almost thought I lost my Yubikey pin when signing the certs...
- I really REALLY need to get a DNS server going... getting tired of updating host files (or maybe I'll just slap something together with an Ansible playbook)
- Networking used to be simple, now here I am dealing with CIDRs and BGP on my poor home network...
- Honestly I think it's a lot because development is very much steered by people with ridiculously complex network topologies where they actually know the ASN number of their router
- You won't believe how many browser tabs went into this
- NFS was actually not as bad as I thought