Use Google Kubernetes Engine with an on-premises non-internet facing Docker registry
Published: 2020-06-11 17:54:31

This guide describes how to connect to a private on-premises non-internet facing Docker registry from a [GKE](https://cloud.google.com/kubernetes-engine) cluster. In order to demonstrate how to do that, we'll use two separate [GCP](https://cloud.google.com/) networks, one for the application cluster, and one for the private registry. Similarely, the latter can be replaced by your own on-premises network.

To follow this tutorial, you need to have a knowledge of the followings:

- General networking knowledges: [RFC 1918](https://tools.ietf.org/html/rfc1918), routers, ...
- [Networking with Google Cloud](https://cloud.google.com/vpc/docs/vpc)
- [Kubernetes](https://kubernetes.io/)
- [GKE](https://cloud.google.com/kubernetes-engine)
- [Docker](https://www.docker.com/)

The tutorial will take you through all the following steps:

- Provisioning the two networks.
- Connecting the networks using Cloud Routers and VPN tunnels.
- Creating the GKE cluster in the on-premises network and deploying a private Docker registry without exposing it to the internet.
- Creating the GKE cluster in the GCP network and deploying a container image uploaded to the private Docker registry.

## Caveats

You will learn how to use [Cloud VPN](https://cloud.google.com/vpn/docs/concepts/overview) to connect your on-premises network to your GCP network. Although, you have to consider the limitations and the performance impact of using this approach.

Because the application GKE cluster will regularly download images from the on-premises network, it can have a significant impact on the trafic going through the VPN tunnels. If you need a higher network bandwidth, please review the [Google Cloud hybrid connectivity documentation](https://cloud.google.com/hybrid-connectivity?) alternatives to Cloud VPN.

If possible, consider using [GCR](https://cloud.google.com/container-registry) for private hosting of container images. GCR is the container registry of choice for using with GKE. It works as a fully managed solution and it comes with built-in vulnerability scanning for the hosted images.

Also because of its very [reasonable cost](https://cloud.google.com/container-registry/pricing), GCR might be a cheaper solution than leveraging [Cloud VPN](https://cloud.google.com/vpn/pricing).

## Before you begin

You need a GCP project.
The fastest way to get one is to connect or create an account on the [GCP web console](http://console.cloud.google.com/).

You'll need to create a new billing account and associate it to the project. For new comers, Google offers free credits to help you getting started at no cost. Please review the documentation for using the [GCP free trial](https://cloud.google.com/free/docs/gcp-free-tier#free-trial).

Also make sure to run the following commands.

Configure the default GCP project.

```bash
export GPC_PROJECT_ID=<your GCP project id>
gcloud config set project $GPC_PROJECT_ID
```

Activate the required APIs.

```bash
gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable dns.googleapis.com
gcloud services enable storage-api.googleapis.com
```

## Provisioning the networks

In this section, we create the two Google [VPCs](https://cloud.google.com/vpc) and create one subnet in western europe for each one of them.

1) We first create the network that will mimic the on-premises network.

```bash
gcloud compute networks create on-prem \
  --subnet-mode=custom \
  --bgp-routing-mode=regional
```

2) And create a subnet in region `europe-west1`.

```bash
gcloud compute networks subnets create on-prem-europe-west1 \
  --range=10.0.0.0/24 \
  --network=on-prem \
  --region=europe-west1
```

3) Then we create the second network, that will have the application GKE cluster.

```bash
gcloud compute networks create app \
  --subnet-mode=custom \
  --bgp-routing-mode=regional
```

4) And create a subnet in region `europe-west1` for this network as well.

```bash
gcloud compute networks subnets create app-europe-west1 \
  --range=10.0.1.0/24 \
  --network=app \
  --region=europe-west1
```

5) Then we add firewall rules in both networks to allow ICMP protocol.

```bash
gcloud compute firewall-rules create on-prem-allow-icmp --direction=INGRESS --priority=1000 --network=on-prem --action=ALLOW --rules=icmp --source-ranges=0.0.0.0/0

gcloud compute firewall-rules create app-allow-icmp --direction=INGRESS --priority=1000 --network=app --action=ALLOW --rules=icmp --source-ranges=0.0.0.0/0
```

## Connecting the networks

In this section, we provide a VPN gateway in each network and use [Cloud Router to create BGP sessions](https://cloud.google.com/router/docs/how-to/configuring-bgp) and dynamically exchange routes between the two networks.

1) First we reserve two regional IPv4 addresses, one for the on-premises VPN gateway and one for the GCP network VPN gateway.

```bash
gcloud compute addresses create app-vpn-ip --region=europe-west1
gcloud compute addresses create on-prem-vpn-ip --region=europe-west1
```

We'll keep the addresses in environment variables for later use.

```bash
APP_VPN_IP=$(gcloud compute addresses describe app-vpn-ip --region europe-west1 --format="value(address)")
ON_PREM_VPN_IP=$(gcloud compute addresses describe on-prem-vpn-ip --region europe-west1 --format="value(address)")
```

2) Then we create the routers, one for each network.

```bash
gcloud compute routers create on-prem-router \
  --region=europe-west1 \
  --network=on-prem \
  --asn=64512 # any private ASN that you are not using elsewhere in your network https://fr.wikipedia.org/wiki/Autonomous_System

gcloud compute routers create app-router \
  --region=europe-west1 \
  --network=app \
  --asn=64513 # needs to be different from on-prem-router ASN
```

3) Then we create the on-premises VPN.

Create the VPN gateway.

```bash
gcloud compute target-vpn-gateways create "on-prem-vpn" \
  --region "europe-west1" \
  --network "on-prem"
```

Then the required forwarding rules.

```bash
gcloud compute forwarding-rules create "on-prem-vpn-rule-esp" \
  --region "europe-west1" \
  --address $ON_PREM_VPN_IP \
  --ip-protocol "ESP" \
  --target-vpn-gateway "on-prem-vpn"

gcloud compute forwarding-rules create "on-prem-vpn-rule-udp500" \
  --region "europe-west1" \
  --address $ON_PREM_VPN_IP \
  --ip-protocol "UDP" \
  --ports "500" \
  --target-vpn-gateway "on-prem-vpn"

gcloud compute forwarding-rules create "on-prem-vpn-rule-udp4500" \
  --region "europe-west1" \
  --address $ON_PREM_VPN_IP \
  --ip-protocol "UDP" \
  --ports "4500" \
  --target-vpn-gateway "on-prem-vpn"
```

Then the VPN tunnel.

```bash
gcloud compute vpn-tunnels create "on-prem-vpn-tunnel-1" \
  --region "europe-west1" \
  --peer-address $APP_VPN_IP \
  --shared-secret "supersecretkey" \
  --ike-version "2" \
  --target-vpn-gateway "on-prem-vpn" \
  --router "on-prem-router" \
  --router-region "europe-west1"
```

And finally the BGP session for dynamic routing.

```bash
gcloud compute routers add-interface on-prem-router \
  --interface-name "bgp-to-app" \
  --vpn-tunnel "on-prem-vpn-tunnel-1" \
  --vpn-tunnel-region "europe-west1" \
  --ip-address "169.254.0.1" \
  --mask-length 16

gcloud compute routers add-bgp-peer on-prem-router \
  --peer-asn 64513 \
  --peer-name "bgp-to-app" \
  --interface "bgp-to-app" \
  --peer-ip-address "169.254.0.2"
```

4) We create a second VPN in the other network.

```bash
gcloud compute target-vpn-gateways create "app-vpn" \
  --region "europe-west1" \
  --network "app"

gcloud compute forwarding-rules create "app-vpn-rule-esp" \
  --region "europe-west1" \
  --address $APP_VPN_IP \
  --ip-protocol "ESP" \
  --target-vpn-gateway "app-vpn"

gcloud compute forwarding-rules create "app-vpn-rule-udp500" \
  --region "europe-west1" \
  --address $APP_VPN_IP \
  --ip-protocol "UDP" \
  --ports "500" \
  --target-vpn-gateway "app-vpn"

gcloud compute forwarding-rules create "app-vpn-rule-udp4500" \
  --region "europe-west1" \
  --address $APP_VPN_IP \
  --ip-protocol "UDP" \
  --ports "4500" \
  --target-vpn-gateway "app-vpn"

gcloud compute vpn-tunnels create "app-vpn-tunnel-1" \
  --region "europe-west1" \
  --peer-address $ON_PREM_VPN_IP \
  --shared-secret "supersecretkey" \
  --ike-version "2" \
  --target-vpn-gateway "app-vpn" \
  --router "app-router" \
  --router-region "europe-west1"

gcloud compute routers add-interface app-router \
  --interface-name "bgp-to-on-prem" \
  --vpn-tunnel "app-vpn-tunnel-1" \
  --vpn-tunnel-region "europe-west1" \
  --ip-address "169.254.0.2" \
  --mask-length 16

gcloud compute routers add-bgp-peer app-router \
  --peer-asn 64512 \
  --peer-name "bgp-to-on-prem" \
  --interface "bgp-to-on-prem" \
  --peer-ip-address "169.254.0.1"
```

When everything is set, you should be able to see these two routes by navigating to VPC Networks > Routes > DYNAMIC.

![Dynamic routes](https://dev-to-uploads.s3.amazonaws.com/i/4uhjosopyvsa0okrz0iu.png)

## Deploying a Docker registry

In this section, we create a new GKE cluster named `artifactory-cluster` in the on-premises network and deploy a non-internet facing [Artifactory](https://jfrog.com/artifactory) in it.
We'll use a self signed SSL certificate for serving HTTPS trafic. Finally we'll create a repository using the Artifactory web interface and publish a Docker image to it.

1) First we create the GKE cluster.

```bash
gcloud beta container clusters create "artifactory-cluster" \
  --zone "europe-west1-b" \
  --no-enable-basic-auth \
  --release-channel "regular" \
  --machine-type "n1-standard-4" \
  --image-type "COS" \
  --disk-type "pd-standard" \
  --disk-size "100" \
  --num-nodes "3" \
  --enable-private-nodes \
  --master-ipv4-cidr "172.16.0.0/28" \
  --enable-ip-alias \
  --network "projects/$GPC_PROJECT_ID/global/networks/on-prem" \
  --subnetwork "projects/$GPC_PROJECT_ID/regions/europe-west1/subnetworks/on-prem-europe-west1" \
  --enable-master-authorized-networks \
  --master-authorized-networks 0.0.0.0/0
```

2) Then we use Cloud NAT to create a gateway to the internet for private IP only VMs in the on-premises network. This is required for the nodes to be able to pull the public Artifactory Docker images.

```bash
gcloud compute routers nats create on-prem-nat-gw \
  --router "on-prem-router" \
  --region "europe-west1" \
  --auto-allocate-nat-external-ips \
  --nat-all-subnet-ip-ranges
```

3) Then we use openssl to generate a certificate authority (CA) key pair.

```bash
export ARTIFACTORY_PKI=/tmp/pki

mkdir $ARTIFACTORY_PKI

openssl genrsa -out "$ARTIFACTORY_PKI/ca.key" 2048

openssl req \
  -x509 \
  -new \
  -nodes \
  -key "$ARTIFACTORY_PKI/ca.key" \
  -subj "/CN=my-ca" \
  -days 10000 \
  -out "$ARTIFACTORY_PKI/ca.crt"
```

4) Then we use it to generate and sign a key pair for the Artifactory HTTPS configuration.

```bash
openssl genrsa \
  -out "$ARTIFACTORY_PKI/tls.key" \
  2048

openssl req \
  -new \
  -key "$ARTIFACTORY_PKI/tls.key" \
  -subj "/CN=artifactory.local" \
  -out "$ARTIFACTORY_PKI/tls.csr"

openssl x509 \
  -req \
  -in "$ARTIFACTORY_PKI/tls.csr" \
  -CA "$ARTIFACTORY_PKI/ca.crt" \
  -CAkey "$ARTIFACTORY_PKI/ca.key" \
  -CAcreateserial \
  -out "$ARTIFACTORY_PKI/tls.crt" \
  -days 365
```

> Note the common name (CN) `artifactory.local`, we'll need to register a local DNS record for this domain in the app network.

5) Then we install Artifactory by configuring and using kubectl

```bash
gcloud container clusters get-credentials artifactory-cluster --zone "europe-west1-b"

kubectl create secret tls artifactory-tls-secret \
  --key "$ARTIFACTORY_PKI/tls.key" \
  --cert "$ARTIFACTORY_PKI/tls.crt"

kubectl apply -f "https://raw.githubusercontent.com/aubm/gke-private-registry-resources/master/artifactory.yaml"
```

After having waited a few minutes for all the containers to be ready, running `kubectl get pods` should give the following output.

```
NAME                                 READY   STATUS    RESTARTS   AGE
artifactory-0                        1/1     Running   0          9m
artifactory-nginx-7fdc5fcb77-5qmq9   1/1     Running   0          9m
postgresql-0                         1/1     Running   0          9m
```

6) Using a web browser, we connect to the Artifactory web interface and go through the welcome wizard.
The default user is `admin` and the password is `password`.
We can skip most steps, except for the admin password reset form. We'll set a new password and remember it for later.

> Discover the Artifactory public IP by running `kubectl get services artifactory-nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`. This IP is only provided for ease of Artifactory configuration through the web interface. For pulling Docker image from the GKE application cluster, we'll use an internal load balancer private IP address.

7) When completed the configuration, we navigate to Administration > Repositories, and click _New Local Repository_. The URL path is `/ui/admin/repositories/local/new`. We enter the name `stack-labs` in the _Repository Key_ field, leave the rest to the defaults, and click _Save & Finish_.

![Create a new local repository](https://dev-to-uploads.s3.amazonaws.com/i/96rea3reddh49jai92jr.png)

## Uploading an image on the registry

In this section, we pull a public image from [Docker Hub](https://hub.docker.com/), and push it to our Artifactory.

1) Because TLS is configured using a self signed SSL certificate, we first need to tell our local Docker daemon that it can trust our registry. We do that by adding a new entry for `artifactory.local` in the `/etc/hosts` file, and then copying the CA certificate file in a specific location. Where exactly is that location depends on your operating system, please refer to the [Docker documentation](https://docs.docker.com/registry/insecure/#use-self-signed-certificates). Here is how to do it when using Docker for Mac.

```bash
export ARTIFACTORY_PUBLIC_IP=$(kubectl get services artifactory-nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo "$ARTIFACTORY_PUBLIC_IP artifactory.local" >> /etc/hosts

mkdir -p "$HOME/.docker/certs.d/artifactory.local"

cp "$ARTIFACTORY_PKI/ca.crt" "$HOME/.docker/certs.d/artifactory.local/ca.crt"

# Restart Docker daemon
```

2) Then we login to the registry.

```bash
docker login artifactory.local --username admin --password <YOUR_PASSWORD>
```

3) And finally pull the `nginx:stable` image, tag it and push it to the Artifactory.

```bash
docker pull nginx:stable

docker tag nginx:stable artifactory.local/stack-labs/nginx:stable

docker push artifactory.local/stack-labs/nginx:stable
```

4) We can confirm that the image is visible by navigating the Artifactory web UI to JFrog Container Registry > Artifacts.

![Uplaoded Nginx image](https://dev-to-uploads.s3.amazonaws.com/i/7x2o1t69fkxkcz12ibx3.png)

## Using the registry

In this section, we create a second GKE cluster in the `app` network, and configure a pod to pull the nginx image that we just pushed to the local Artifactory. We do that by connecting using only the local interface instead of the public load balancer IP address. For that, we create a new private DNS zone using [Cloud DNS](https://cloud.google.com/dns) and add a new record to associate the private IP address to the `artifactory.local` domain.

1) First we create the private DNS zone in the `app` network.

```bash
gcloud dns managed-zones create artifactory-local \
  --dns-name "artifactory.local" \
  --description "DNS for artifactory.local" \
  --visibility "private" \
  --networks "app"
```

2) Then we add the DNS record.

```bash
export ARTIFACTORY_PRIVATE_IP=$(kubectl get services artifactory-nginx-internal -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

gcloud dns record-sets transaction start --zone "artifactory-local"

gcloud dns record-sets transaction add $ARTIFACTORY_PRIVATE_IP \
  --name "artifactory.local." \
  --ttl 60 \
  --type "A" \
  --zone "artifactory-local"

gcloud dns record-sets transaction execute --zone "artifactory-local"
```

3) Then we create the application GKE cluster.

```bash
gcloud beta container clusters create "application-cluster" \
  --zone "europe-west1-b" \
  --no-enable-basic-auth \
  --release-channel "regular" \
  --machine-type "n1-standard-1" \
  --image-type "COS" \
  --disk-type "pd-standard" \
  --disk-size "100" \
  --num-nodes "1" \
  --enable-private-nodes \
  --master-ipv4-cidr "172.17.0.0/28" \
  --enable-ip-alias \
  --network "projects/$GPC_PROJECT_ID/global/networks/app" \
  --subnetwork "projects/$GPC_PROJECT_ID/regions/europe-west1/subnetworks/app-europe-west1" \
  --enable-master-authorized-networks \
  --master-authorized-networks 0.0.0.0/0
```

4) Then, we need to configure the GKE nodes in a way that they can trust the registry's self signed certificate. The below commands:

- Create a new secret that contains the CA certificate.
- Create a NAT gateway for the nodes to be able to pull public images from Docker Hub.
- Deploy a privileged daemonset that mounts the secret and copies its contents to the host file system on `/host/etc/docker/certs.d/artifactory.local/ca.crt`. The daemonset uses a public Alpine image, which is why the NAT gateway is necessary. More on this solution [here](https://cloud.google.com/solutions/automatically-bootstrapping-gke-nodes-with-daemonsets).

```bash
kubectl create secret generic init-pki --from-file="ca.crt=$ARTIFACTORY_PKI/ca.crt"

gcloud compute routers nats create app-nat-gw \
  --router "app-router" \
  --region "europe-west1" \
  --auto-allocate-nat-external-ips \
  --nat-all-subnet-ip-ranges

kubectl apply -f "https://raw.githubusercontent.com/aubm/gke-private-registry-resources/master/init-daemonset.yaml"
```

> You can avoid this step by configuring the Docker registry with a certificate signed by a well known authority.

5) Then we create a Kubernetes secret containing the Artifactory Docker registry credentials, by reconfiguring and using kubectl.

```bash
gcloud container clusters get-credentials application-cluster --zone "europe-west1-b"

kubectl create secret docker-registry artifactory-local \
  --docker-server "artifactory.local" \
  --docker-username "admin" \
  --docker-password "<the password that you changed from the Artifactory web UI>"
```

6) Finally we run the nginx pod that uses the previously defined secret. More on pulling images from private registries in the [Kubernetes documentation](https://kubernetes.io/fr/docs/tasks/configure-pod-container/pull-image-private-registry/).

```bash
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      imagePullSecrets:
      - name: artifactory-local
      containers:
      - image: artifactory.local/stack-labs/nginx:stable
        name: nginx
        resources: {}
status: {}
EOF
```

## Cleaning up

```bash
gcloud container clusters delete application-cluster --zone "europe-west1-b" --quiet
gcloud container clusters delete artifactory-cluster --zone "europe-west1-b" --quiet

gcloud compute routers nats delete app-nat-gw --router "app-router" --region "europe-west1" --quiet
gcloud compute routers nats delete on-prem-nat-gw --router "on-prem-router" --region "europe-west1" --quiet

gcloud beta dns record-sets transaction start --zone=artifactory-local
gcloud beta dns record-sets transaction remove $ARTIFACTORY_PRIVATE_IP --name=artifactory.local. --ttl=60 --type=A --zone=artifactory-local
gcloud beta dns record-sets transaction execute --zone=artifactory-local
gcloud dns managed-zones delete artifactory-local

gcloud compute routers remove-bgp-peer app-router --peer-name "bgp-to-on-prem"
gcloud compute routers remove-interface app-router --interface-name "bgp-to-on-prem"
gcloud compute vpn-tunnels delete "app-vpn-tunnel-1" --region "europe-west1" --quiet
gcloud compute forwarding-rules delete "app-vpn-rule-udp4500" --region "europe-west1" --quiet
gcloud compute forwarding-rules delete "app-vpn-rule-udp500" --region "europe-west1" --quiet
gcloud compute forwarding-rules delete "app-vpn-rule-esp" --region "europe-west1" --quiet
gcloud compute target-vpn-gateways delete "app-vpn" --region "europe-west1" --quiet

gcloud compute routers remove-bgp-peer on-prem-router --peer-name "bgp-to-app"
gcloud compute routers remove-interface on-prem-router --interface-name "bgp-to-app"
gcloud compute vpn-tunnels delete "on-prem-vpn-tunnel-1" --region "europe-west1" --quiet
gcloud compute forwarding-rules delete "on-prem-vpn-rule-udp4500" --region "europe-west1" --quiet
gcloud compute forwarding-rules delete "on-prem-vpn-rule-udp500" --region "europe-west1" --quiet
gcloud compute forwarding-rules delete "on-prem-vpn-rule-esp" --region "europe-west1" --quiet
gcloud compute target-vpn-gateways delete "on-prem-vpn" --region "europe-west1" --quiet

gcloud compute routers delete on-prem-router --region=europe-west1 --quiet
gcloud compute routers delete app-router --region=europe-west1 --quiet

gcloud compute addresses delete app-vpn-ip --region=europe-west1 --quiet
gcloud compute addresses delete on-prem-vpn-ip --region=europe-west1 --quiet

gcloud compute firewall-rules delete on-prem-allow-icmp --quiet
gcloud compute firewall-rules delete app-allow-icmp --quiet

gcloud compute networks subnets delete on-prem-europe-west1 --region=europe-west1 --quiet
gcloud compute networks delete on-prem --quiet

gcloud compute networks subnets delete app-europe-west1 --region=europe-west1 --quiet
gcloud compute networks delete app --quiet
```