Virtual Kubernetes Clusters: What Are They Needed For?
Tags: devops, kubernetes, productivity, k3s
Published: 2022-08-29 18:58:50

##Developer Wishlist Never Ends

Imagine you can
- have **many** virtual clusters **within a single** cluster, and 
- they are **much cheaper** than the traditional Kubernetes clusters, and 
- they require **lower** management and maintenance **efforts**. 

Sounds intriguing, eh? This makes v/clusters ideal for running experiments, continuous integration, and setting up **sandbox** 🧪 environments.

So, Loft Labs created such a solution written natively in Golang and made it an ~2k⭐ [open source](https://github.com/loft-sh/vcluster).

##What's under the hood?

On top of other Kubernetes clusters, virtual clusters are fully functional Kubernetes clusters. Virtual clusters utilize the worker nodes and networking of the host cluster, as opposed to completely distinct "real" clusters. They schedule all workloads into a single namespace of the host cluster and have their own control plane. Virtual clusters divide a single physical cluster into several distinct ones, much like virtual machines do.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1k34razhtr4v2t1dp3n8.png)
🖼️ Right click, don't even think too long.

Only the essential Kubernetes components - the API server, controller manager, storage backend (such as etcd, sqlite, mysql, etc.), and - optionally - a scheduler—make up the virtual cluster itself. In order to minimize virtual cluster overhead, `vcluster` builds by default on `k3s`, a fully functional, certified, lightweight Kubernetes distribution that compiles the Kubernetes components into a single binary and disables by default all unnecessary Kubernetes features, such as the pod scheduler or specific controllers.

Other Kubernetes distributions, [such k0s and vanilla k8s](https://www.vcluster.com/docs/operator/other-distributions), are supported in addition to k3s. In addition to the control plane, the virtual cluster also includes a Kubernetes hypervisor that simulates networking and worker nodes. Between the virtual and host clusters, this component syncs a few key resources that are crucial for cluster functionality:
- **Pods**: All the virtual cluster's started pods are rewritten before being launched in the virtual cluster's namespace in the host cluster. Environment variables, DNS, service account tokens, and other configurations are updated to point to the virtual cluster rather than the host cluster. In the pod, it appears that the virtual cluster rather than the host cluster is where the pod is started.
- **Services**: On the namespace of the virtual cluster in the host cluster, all services and endpoints are rewritten and generated. The service cluster IPs are shared by the host cluster and virtual cluster. This implies that there are no performance consequences when a service in the host cluster is accessed from within the virtual cluster.
- **PersistentVolumeClaims**: In the event that persistent volume claims are generated in the virtual cluster, they will be modified and generated in the host cluster's namespace. The relevant persistent volume data will be synchronized back to the virtual cluster if they are bound in the host cluster.
- **ConfigMaps & Secrets**: Only ConfigMaps and secrets mounted to pods within the virtual cluster will be synced to the host cluster; all other ConfigMaps and secrets will only be retained within the virtual cluster.
- **Other Resources**: Deployments, StatefulSets, CRDs, service accounts, etc. do not sync with the host cluster; instead, they only reside in the virtual cluster.

##Who lost the magic mirror?
For each pod with the `spec.nodeName` value it encounters inside the virtual cluster, `vcluster` by default creates a _false_ node. Because `vcluster` _does not_ by default _have RBAC permissions_ to access the real nodes in the host cluster because doing so would require a cluster role and cluster role binding, those _false_ nodes are produced. Additionally, each node will get a false `kubelet` endpoint that will either send requests to the real node _or rewrite them_ to keep virtual cluster names intact.

Vcluster supports multiple modes to customize node syncing behavior. For a detailed list of the resources that may have been synced, [see details here](https://www.vcluster.com/docs/architecture/synced-resources) in the docs.

The hypervisor also proxies some Kubernetes API calls, including pod port forwarding or container command execution, to the host cluster in addition to synchronizing virtual and host cluster resources. It essentially performs the function of the virtual cluster's reverse proxy.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iay4g5cnqmdlgak8mxih.png)

To ensure proper network operation for the virtual cluster, resources like `Service` and `Ingress` are synced by default _from_ the virtual cluster [down] _to_ the host cluster.

##There are never too many levels of abstraction

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4inkwzd1ss3rnja32wie.png)

Certain resources (such as CRDs or RBAC policies) reside _cluster-wide_, and you can’t isolate them using _namespaces_. For instance, it is not feasible to install an operator simultaneously in multiple versions inside a same cluster.

```
$ kubectl api-resources --namespaced=false|true
```

Although Kubernetes itself already offers namespaces for various settings, their use of cluster-scoped resources and the control plane is constrained.

In many circumstances, virtual clusters are also more stable than namespaces. In its own data store, the virtual cluster produces its own Kubernetes resource objects. These resources are unknown to the host cluster.

This kind of isolation is good for resilience. The necessity for access to cluster-scoped resources like cluster roles, shared CRDs, or persistent volumes still exists for engineers who adopt namespace-based isolation. Each team that depends on one of these shared resources will probably experience failure if an engineer destroys something in it.

Finally, virtual cluster configuration _is independent of physical_ cluster configuration. This is excellent for multi-tenancy because it allows you to easily create a fresh environment or amazing demo applications. 😎

##How it looks in your CLI
Create file `vcluster.yaml`:

```yaml
vcluster:
  image: rancher/k3s:v1.23.5-k3s1   # Choose k3s version
```
Then, install helm chart using vcluster.yaml for chart values:

```
helm upgrade --install my-vcluster vcluster \
  --values vcluster.yaml \
  --repo https://charts.loft.sh \
  --namespace host-namespace-1 \
  --repository-config=''
```
###Access:

Get the admin tool
```
curl -s -L "https://github.com/loft-sh/vcluster/releases/latest" | sed -nE 's!.*"([^"]*vcluster-linux-amd64)".*!https://github.com\1!p' | xargs -n 1 curl -L -o vcluster && chmod +x vcluster;
sudo mv vcluster /usr/local/bin;
```

Then, connect:
```
# Connect and switch the current context to the vcluster
vcluster connect my-vcluster -n my-vcluster

# Switch back context
vcluster disconnect
```

You have an option to create a separate `kubeconfig` to use instead of changing the current context:
```
vcluster connect my-vcluster --update-current=false
```
Or you may execute a command directly with `vcluster` context without changing the _current_ context:
```
vcluster connect my-vcluster -- kubectl get namespaces
vcluster connect my-vcluster -- bash
```

###Usage:
```
# Run any kubectl, helm, etc. command in your vcluster
kubectl get namespace
kubectl get pods -n kube-system
kubectl create namespace demo-nginx
kubectl create deployment nginx-deployment -n demo-nginx --image=nginx
kubectl get pods -n demo-nginx
```
###Cleanup:
```
helm delete my-vcluster -n vcluster-my-vcluster --repository-config=''
```

##What if you're planning some serious thing?

Well, the stock K8s distribution _is compatible with high availability_ in `vcluster`. What is meant by high availability? Well, one of the entities is to make `etcd` database more robust. The second one is to boost syncer's performance. As mentioned above, `vcluster` uses a so-called syncer which copies the pods that are created within the virtual cluster to the underlying host cluster. 

🪲**TL;DR #1:** `etcd` uses a leader-based consensus protocol for consistent data replication and log execution. Etcd cluster members elect a single leader, all other members become followers. The cluster elects a new leader automatically when one falls out of favor. Once the incumbent fails, the election does not take place immediately. Since the failure detection methodology is timeout based, electing a new leader takes roughly an election timeout. 

🪲**TL;DR #2:** Why it is recommended to have _a minimum of three instances_ in an etcd cluster, is [well described here](https://etcd.io/docs/v3.5/faq/), a first hand info.

Currently, vcluster's high availability setup does not allow single binary distributions like `k0s` and `k3s`.

Create a values.yaml with the following structure in order to operate `vcluster` in high availability mode:

```yaml
# Enable HA mode
enableHA: true

# Scale up syncer replicas
syncer:
  replicas: 3

# Scale up etcd
etcd:
  replicas: 3

# Scale up controller manager
controller:
  replicas: 3

# Scale up api server
api:
  replicas: 3

# Scale up DNS server
coredns:
  replicas: 3
```

##To summarize
A virtual Kubernetes cluster that is fully functioning can be built using `vcluster`! The underlying K8s cluster's namespace is where each `vcluster` operates. It provides better multi-tenancy and isolation than conventional namespaces, and it is less expensive than building independent, fully-fledged clusters.

Virtual clusters can be a good option to running numerous instances of `k3s`, or `k0s` side by side, but they _cannot exist on their own without a host_ cluster. 

Compared to fully independent Kubernetes clusters, they are faster, lighter, and simpler to reach. Therefore, give using a virtual cluster a shot [if you're tired](https://komodor.com/learn/git-revert-rolling-back-in-gitops-and-kubernetes/) of having to reset your local or CI/CD Kubernetes clusters all the time. However, this is a topic for a completely different story, much sadder than what you just read.

Be in good & non-ghost shape! 👻

Many thanks to Viktor 🐦@vfarcic Farcic and Mauricio 🐦@salaboy Salatino for inspiration!