Deploy Preview Environments to Any Kubernetes Cluster with Preevy
Tags: kubernetes, devops, webdev, opensource
Published: 2023-07-11 11:39:51

## What?

We're excited to announce that [Preevy](preevy.dev) can now deploy your preview environments to any Kubernetes cluster! Kubernetes support is a major improvement to Preevy, and a big step in our journey to include and empower a broader community of developers.

## Why?

Kubernetes-powered ephemeral preview environments are faster, scalable and cost-effective alternative to VMs.

While preview environments live for the duration of a Pull Request, they will typically only see little bursts of actual usage. Kubernetes is a great way to oversubsrcibe compute resources. Environments can be configured to require little CPU and memory while idle and waiting for a review.

With Kubernetes, there's no need to provision VMs. Instead, a single Pod is provisioned for every environment. This is about as fast as starting a [Docker-in-Docker](https://hub.docker.com/_/docker) container.

## How?

Preevy creates ephemeral preview environments by deploying your Docker Compose project on a server, and exposing it to the internet (or your private network) via a [proxy service](https://livecycle.io/blogs/preevy-proxy-service-2/). Before Kubernetes support, Preevy needed to provision a VM with a Docker server for each environment. With Kubernetes, it provisions a Docker server as a Kubernetes Pod and runs your project inside it. Your services are still exposed using the Preevy Tunnel Service.

![1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zuz7k5rwank4j82gdulb.png)

To start using Preevy with Kubernetes today, follow our [Getting Started](https://preevy.dev/#getting-started) guide and see the [Kubernetes Driver docs](https://preevy.dev/drivers/kube-docker).

### Preevy's driver contract

Drivers are Preevy's way to offer an extensible framework for deploying environments on a variety of providers. The first driver we released was the [AWS Lightsail driver](https://preevy.dev/drivers/aws-lightsail), which deploys your environment on a budget-friendly VM on Amazon's cloud. Since then, we've added support for [Google Cloud](https://preevy.dev/drivers/gcp-gce) and [Microsoft Azure]().

Common to all the VM drivers is the ability to connect to the provisioned machines using SSH.
Once a VM exists, SSH is used to copy any volume files, and finally to establish a port-forward session to the remote Docker Server in order to run the `docker compose up` command.

So it was only natural to have a SSH-based contract: A `createMachine` function which creates a VM (installing Docker if needed), and returns the machine's public IP, an SSH keypair and a username.

![2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0o2wxtik8b57jkv90sqo.png)

SSH makes less sense when deploying to Kubernetes - while it is possible to SSH into a Pod, it requires setting up an ingress and exposing the SSH port publicly, which is impracticle for most installations. Instead, we had to redefine the driver's contract to create a higher-level abstraction: `createMachine` now returns a *"connection"* object which can do the following:

- Execute a command - used to copy files (by running `tar` with a piped STDIN)
- Port-forward to the Docker Server

![3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b6y26kr1k4kw94vp3hqe.png)

In the Kubernetes driver, the *"connection"* functionality is implemented using the Kubernetes [exec](https://kubernetes.io/docs/tasks/debug/debug-application/get-shell-running-container/) and [port-forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) APIs.

### Customizing the Kubernetes resources

The Kubernetes driver introduces [templates](https://preevy.dev/drivers/kube-docer#customizing-the-provisioned-kubernetes-resources), a new mechanism to customize the deployed resources. For example, Kubernetes admins might want to tag the deployed resources using [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) and [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/), maybe to provision them on a specific set of worker nodes. Or, some users will want to provision additional resources per environment, like a database.

To use a custom template, configure it in the [`preevy init` command](https://preevy.dev/#getting-started), or specify the `--kube-pod-template` argument to the [`preevy up` command](https://preevy.dev/cli-reference#preevy-up-service).

Here is an example template:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ id }}-dc
  namespace: {{ namespace }}
data:
  daemon.json: |
    {
      "tls": false
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ id }}
  namespace: {{ namespace }}
  labels:
    app: preevy-{{ id }}
    app.kubernetes.io/component: docker-host
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  runtimeClassName: sysbox-runc
  selector:
    matchLabels:
      app: preevy-{{ id }}
  template:
    metadata:
      labels:
        app: preevy-{{ id }}
    spec:
      containers:
      - name: docker
        image: nestybox/alpine-docker
        command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock"]
        volumeMounts:
        - mountPath: /etc/docker
          name: docker-config

      volumes:
      - name: docker-config
        configMap:
          name: {{ id }}-dc
```

This template specifies a custom docker image `nestybox/alpine-docker`, a custom `runtimeClassName` and some annotations to allow running a rootless, non-provisioned Docker Server environment using [Sysbox](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/deploy.md#deploying-pods-with-kubernetes--sysbox).

Here is a another template which deploys a test Kafka broker, [Zookeeper-free](https://github.com/livecycle/kafka-docker), with every environment:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ id }}-dc
  namespace: {{ namespace }}
data:
  daemon.json: |
    {
      "tls": false
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ id }}
  namespace: {{ namespace }}
  labels:
    app: preevy-{{ id }}
    app.kubernetes.io/component: docker-host
spec:
  replicas: 1
  selector:
    matchLabels:
      app: preevy-{{ id }}
  template:
    metadata:
      labels:
        app: preevy-{{ id }}
    spec:
      containers:
      - name: docker
        image: docker:24-dind
        securityContext:
          privileged: true
        command: ["dockerd", "--host=tcp://0.0.0.0:2375", "--host=unix:///var/run/docker.sock"]
        volumeMounts:
        - mountPath: /etc/docker
          name: docker-config
        env:
          - name: KAFKA_SERVICE
            value: {{ id }}-kafka
      volumes:
      - name: docker-config
        configMap:
          name: {{ id }}-dc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka
  namespace: {{ namespace }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{ id }}-kafka
  template:
    metadata:
      labels:
        app: {{ id }}-kafka
    spec:
      enableServiceLinks: false
      containers:
      - name: kafka-broker
        image: ghcr.io/livecycle/kafka-docker:lc-1.1.0
        ports:
        - containerPort: 9092
          name: kafka
          protocol: TCP
        env:
          - name: KAFKA_WITHOUT_ZOOKEEPER
            value: 'true'
          - name: KAFKA_ADVERTISED_LISTENERS
            value: PLAINTEXT://kafka:9092
          - name: KAFKA_LISTENERS
            value: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
          - name: KAFKA_CONTROLLER_QUORUM_VOTERS
            value: 1\@localhost:9093
          - name: CLUSTER_UUID
            value: p8fFEbKGQ22B6M_Da_vCBw
        readinessProbe:
          failureThreshold: 10
          tcpSocket:
            port: 9092
          periodSeconds: 3
          successThreshold: 1
          timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
  name: {{ id }}-kafka
  namespace: {{ namespace }}
spec:
  ports:
  - port: 9092
    protocol: TCP
    targetPort: 9092
  selector:
    app: {{ id }}-kafka
  type: ClusterIP
```

This adds another Deployment and Service to the [default template](https://github.com/livecycle/preevy/blob/main/packages/driver-pod-docker/static/default-template.yaml.njk).

Note the use of the environment variable `KAFKA_SERVICE` to make the Kubernetes Kafka service hostname (suffixed with the unique environment ID) available to the environment at [build time](https://docs.docker.com/compose/environment-variables/set-environment-variables/#substitute-from-the-shell).

## Summary

With Preevy and Kubernetes working together, you can now provision lightweight environments with less effort, reduced costs and tighter integration with your existing infrastructure. Upgrade to the [latest version](https://www.npmjs.com/package/preevy) and give it a try!