Try Kubernetes with microk8s
Tags: kubernetes, tutorial, webdev, beginners
Published: 2022-09-11 13:59:11

## Objective

At the Japan Virtual Summit 2021, I gave a session on Kubernetes, which was targeted at those who had an Azure account or IRIS evaluation license key. I'm sure some developers would like to try it out a little more quickly, therefore, this article will introduce the procedure to run IRIS Community Edition on [microk8s](https://microk8s.io/), a lightweight implementation of k8s that can be used in virtual environments.

For your reference, my environment is as follows:

| Purpose | O/S | Host Type | IP |
|---|---|---|---|
| Client PC | Windows10 Pro | Physical Hosts | 192.168.11.5/24 |
| mirok8s environment | Ubuntu 20.04.1 LTS | Virtual host (vmware) on Windows 10 as listed above | 192.168.11.49/24 |

Ubuntu used [ubuntu-20.04.1-live-server-amd64.iso](http://old-releases.ubuntu.com/releases/20.04.1/ubuntu-20.04.1-live-server-amd64.iso)  to install only the minimum server features.

## Overview

This section describes the steps to deploy IRIS Community Edition as a Kubernetes StatefulSet.
For the persistent storage to store IRIS system files and user database externally, we will use microk8s_hostpath or Longhorn.
You can find the code to use [here](https://github.com/IRISMeister/iris_mk8s).

>The preview version of 2021.1 is being used, but when 2021.1 will be officially released, the image name will be changed and it will not work properly without changing the image: value in the yml.

## Installation

Install and start microk8s.

```
$ sudo snap install microk8s --classic --channel=1.20
$ sudo usermod -a -G microk8s $USER
$ sudo chown -f -R $USER ~/.kub
$ microk8s start
$ microk8s enable dns registry storage metallb
  ・
  ・
Enabling MetalLB
Enter each IP address range delimited by comma (e.g. '10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111'):192.168.11.110-192.168.11.130
```

Since you will be asked for the IP range to be assigned to the load balancer, set an appropriate range. In my environment, the CIDR of the host where k8s is running is 192.168.11.49/24, so I designated [192.168.11.110-192.168.11.130] as the appropriate free IP range.

At this point, the single-node k8s environment is ready.
```
$ microk8s kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
ubuntu 
```


In the following examples, microk8s is omitted. It is troublesome to add microk8s every time you run kubectl, so I configured alias with the following command. In the following examples, microk8s is omitted.
```
$ sudo snap alias microk8s.kubectl kubectl
$ kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
ubuntu   Ready    <none>   10d   v1.20.7-34+df7df22a741dbc
```

>To restore the original state

```
sudo snap unalias kubectl
```

## Launch

```
$ kubectl apply -f mk8s-iris.yml
```

>As IRIS Community Edition, I have not specified a license key nor imagePullSecrets to log in to the container registry.

After a few moments, two pods will be created. IRIS is now up and running

```
$ kubectl get pod
NAME     READY   STATUS    RESTARTS   AGE
data-0   1/1     Running   0          107s
data-1   1/1     Running   0          86s
$ kubectl get statefulset
NAME   READY   AGE
data   2/2     3m32s
$ kubectl get service
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)           AGE
kubernetes   ClusterIP      10.152.183.1     <none>           443/TCP           30m
iris         ClusterIP      None             <none>           52773/TCP         8m55s
iris-ext     LoadBalancer   10.152.183.137   192.168.11.110   52773:31707/TCP   8m55s
```

If the pod's STATUS does not  run, you can check the event with the following command. It may be because the image name is specified incorrectly and the pull fails or because some resource is missing.

```
$ kubectl describe pod data-0
```

The following command will allow you to login to IRIS with O/S authentication.
```
$ kubectl exec -it data-0 -- iris session iris
Node: data-0, Instance: IRIS
USER>
```

## Access the IRIS management portal on individual pods

Use the command below to check the internal IP address of each pod.
```
$ kubectl get pod -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
data-0   1/1     Running   0          46m   10.1.243.202   ubuntu   <none>           <none>
data-1   1/1     Running   0          45m   10.1.243.203   ubuntu   <none>           <none>
```

>Usually, the internal IP cannot be referenced directly from the kubectl executing host (use kubectl port-forward to reference it). Still, the microk8s environment can be accessed this time since they are all running on the same host.

As my virtual environment, Linux does not have a GUI. I can access the management portal from a Windows browser by running the following command on a client PC.

```
C:\temp>ssh -L 9092:10.1.243.202:52773 YourLinuxUserName@192.168.11.49
C:\temp>ssh -L 9093:10.1.243.203:52773 YourLinuxUserName@192.168.11.49
```

> The internal IP will be updated every time the pod is recreated.

| Target | URL | USER | Password |
|---|---|---|---|
| IRIS on pod data-0 | http://localhost:9092/csp/sys/%25CSP.Portal.Home.zen  | SuperUser | SYS |
| IRIS on pod data-1 | http://localhost:9093/csp/sys/%25CSP.Portal.Home.zen  | SuperUser | SYS |

>Password is specified by PasswordHash in CPF

Check the database configuration. The following databases can be confirmed to be created on the PV.

| Database Name | Path |
|---|---|
| IRISSYS | /iris-mgr/IRIS_conf.d/mgr/  |
| TEST-DATA | /vol-data/TEST-DATA/ |

## Stoppage

Delete the created resource.

```
$ kubectl delete -f mk8s-iris.yml --wait
```

Note that this will also delete the IRIS pod, keeping the PV saved.  The next time a pod with the same name is invoked, it will be served the same volume as before. This allows you to separate the life cycle of the pod from the life cycle of the database. You can also delete the PV with the following command (the database contents will be permanently lost):

```
$ kubectl delete pvc --all
```

To shut down the O/S, execute the following command to stop the k8s environment successfully.

```
$ microk8s stop
```

After restarting the O/S, you can start the k8s environment with the following command.
```
$ microk8s start
```
If you want to delete the microk8s environment completely, do the following "before" running microk8s stop. (It took me a long time to do this. I don't think you need to run this daily).
```
$ microk8s reset --destroy-storage
```

## Observation

### Storage location

Out of curiosity, where does /iris-mgr/ exist? microk8s is a standalone k8s environment, so if storageClassName is microk8s-hostpath, the file entity is on the same host. Firstly, use kubectl get pv to check the created PV.

```
$ kubectl apply -f mk8s-iris.yml
$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                               STORAGECLASS        REASON   AGE
pvc-ee660281-1de4-4115-a874-9e9c4cf68083   20Gi       RWX            Delete           Bound    container-registry/registry-claim   microk8s-hostpath            37m
pvc-772484b1-9199-4e23-9152-d74d6addd5ff   5Gi        RWO            Delete           Bound    default/dbv-data-data-0             microk8s-hostpath            10m
pvc-112aa77e-2f2f-4632-9eca-4801c4b3c6bb   5Gi        RWO            Delete           Bound    default/dbv-mgr-data-0              microk8s-hostpath            10m
pvc-e360ef36-627c-49a4-a975-26b7e83c6012   5Gi        RWO            Delete           Bound    default/dbv-mgr-data-1              microk8s-hostpath            9m55s
pvc-48ea60e8-338e-4e28-9580-b03c9988aad8   5Gi        RWO            Delete           Bound    default/dbv-data-data-1             microk8s-hostpath            9m55s
```

Now, we describe default/dbv-mgr-data-0, which is used as the ISC_DATA_DIRECTORY for the data-0 pod.

```
$ kubectl describe pv pvc-112aa77e-2f2f-4632-9eca-4801c4b3c6bb
  ・
  ・
Source:
    Type:   HostPath (bare host directory volume)
    Path:   /var/snap/microk8s/common/default-storage/default-dbv-mgr-data-0-pvc-112aa77e-2f2f-4632-9eca-4801c4b3c6bb
```
This path is the location of the entity file.
```
$ ls /var/snap/microk8s/common/default-storage/default-dbv-mgr-data-0-pvc-112aa77e-2f2f-4632-9eca-4801c4b3c6bb/IRIS_conf.d/
ContainerCheck  csp  dist  httpd  iris.cpf  iris.cpf_20210517  _LastGood_.cpf  mgr
```

> Please, do not use hostpath for storageClassName, because unlike microk8s_hostpath, it will result in multiple IRISs being co-located in the same folder (destroyed state).

### Resolving hostnames

StatefulSet assigns a unique hostname to each pot, such as data-0, data-1, etc., according to the value of metadata.name.
To use this hostname for communication between pots, [Headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-service) is adopted.

```
kind: StatefulSet
metadata:
  name: data

kind: Service
spec:
  clusterIP: None # Headless Service

```

> It is useful when using features such as sharding, which involves communication between nodes. There is no direct benefit in this example.

I want to use nslookup, but the container runtime (ctr) used in kubectl and k8s cannot log in as root like docker. Also, the IRIS container images do not have sudo installed for security reasons, so you cannot use apt install additional software outside of the image build time. Here, we will additionally boot busybox and use nslookup to check the hostname.

```
$ kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
/ # nslookup data-0.iris
Server:    10.152.183.10
Address 1: 10.152.183.10 kube-dns.kube-system.svc.cluster.local

Name:      data-0.iris
Address 1: 10.1.243.202 data-0.iris.default.svc.cluster.local
/ #
```

You will notice that data-0.iris has been assigned an IP address of 10.1.243.202 with an FQDN of data-0.iris.default.svc.cluster.local. 10.152.183.10 is the DNS server provided by k8s. Likewise, data-1.iris is registered in the DNS.

## Using your images

Currently, k8s does not use Docker. Thus, a separate Docker setup is required to build the image.

> k8s is only for the operational environment.

The following is based on the assumption that you have already setup Docker and docker-compose. The image can be any content. Here we will use [simple](https://github.com/IRISMeister/simple)  as an example. This image will provide a straightforward REST service on the MYAPP namespace, where localhost:32000 is the built-in container registry created by microk8s, to which we will push this image.

```
$ git clone https://github.com/IRISMeister/simple.git
$ cd simple
$ ./build.sh
$ docker tag dpmeister/simple:latest localhost:32000/simple:latest
$ docker push localhost:32000/simple:latest
```

> For those who don't want to bother with the build process, the pre-built image is saved as dpmeister/simple:latest, so you can pull it and use it directly as shown below.

```
$ docker pull dpmeister/simple:latest
$ docker tag dpmeister/simple:latest localhost:32000/simple:latest
$ docker push localhost:32000/simple:latest
```

Change the image to localhost:32000/simple by editing yml. In addition, add ModifyNamespace to the cpf action to switch the data storage location from the database in the container (MYAPP-DATA) to an external database (MYAPP-DATA-EXT). An edited file is available as mk8s-simple.yml ( which looks almost the same as mk8s-iris.yml). Use this to run it.

If you already have a pod running, delete it.

```
$ kubectl delete -f mk8s-iris.yml
$ kubectl delete pvc --all
```

```
$ kubectl apply -f mk8s-simple.yml
$ kubectl get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)           AGE
kubernetes   ClusterIP      10.152.183.1     <none>           443/TCP           3h36m
iris         ClusterIP      None             <none>           52773/TCP         20m
iris-ext     LoadBalancer   10.152.183.224   192.168.11.110   52773:30308/TCP   20m

$ curl -s -H "Content-Type: application/json; charset=UTF-8" -H "Accept:application/json" "http://192.168.11.110:52773/csp/myapp/get" --user "appuser:SYS" | python3 -mjson.tool
{
    "HostName": "data-1",
    "UserName": "appuser",
    "Status": "OK",
    "TimeStamp": "05/17/2021 19:34:00",
    "ImageBuilt": "05/17/2021 10:06:27"
}
```

If you repeatedly run curl, the HostName (the hostname on which the REST service ran) may be data-0 or data-1, but this is because it is load balanced (as expected).

> The maximum number of sessions for the Community Edition is 5, which may have been exceeded by some previous action. If this happens, a message will be logged indicating that the license limit has been exceeded.

```
$ kubectl logs data-0
  ・
  ・
05/17/21-19:21:17:417 (2334) 2 [Generic.Event] License limit exceeded 1 times since instance start.
```

## When using Longhorn

> Learn more about distributed Kubernetes storage Longhorn, please click [here](https://www.rancher.co.jp/pdfs/doc/doc-02-Hajimete_Longhorn.pdf).

Run Longhorn and wait until all the pods are READY.

```
$ kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
$ kubectl -n longhorn-system get pods
NAME                                       READY   STATUS    RESTARTS   AGE
longhorn-ui-5b864949c4-72qkz               1/1     Running   0          4m3s
longhorn-manager-wfpnl                     1/1     Running   0          4m3s
longhorn-driver-deployer-ccb9974d5-w5mnz   1/1     Running   0          4m3s
instance-manager-e-5f14d35b                1/1     Running   0          3m28s
instance-manager-r-a8323182                1/1     Running   0          3m28s
engine-image-ei-611d1496-qscbp             1/1     Running   0          3m28s
csi-attacher-5df5c79d4b-gfncr              1/1     Running   0          3m21s
csi-attacher-5df5c79d4b-ndwjn              1/1     Running   0          3m21s
csi-provisioner-547dfff5dd-pj46m           1/1     Running   0          3m20s
csi-resizer-5d6f844cd8-22dpp               1/1     Running   0          3m20s
csi-provisioner-547dfff5dd-86w9h           1/1     Running   0          3m20s
csi-resizer-5d6f844cd8-zn97g               1/1     Running   0          3m20s
csi-resizer-5d6f844cd8-8nmfw               1/1     Running   0          3m20s
csi-provisioner-547dfff5dd-pmwsk           1/1     Running   0          3m20s
longhorn-csi-plugin-xsnj9                  2/2     Running   0          3m19s
csi-snapshotter-76c6f569f9-wt8sh           1/1     Running   0          3m19s
csi-snapshotter-76c6f569f9-w65xp           1/1     Running   0          3m19s
csi-attacher-5df5c79d4b-gcf4l              1/1     Running   0          3m21s
csi-snapshotter-76c6f569f9-fjx2h           1/1     Running   0          3m19s
```

Please modify the storageClassName in all (there are two places) of mk8s-iris.yml to longhorn.
If you are already running on microk8s_hostpath, please delete all pods and PVs, and then follow the above steps. In short...

```
$ kubectl delete -f mk8s-iris.yml --wait
$ kubectl delete pvc --all
   edit mk8s-iris.yml
      before)storageClassName: microk8s-hostpath
      after)storageClassName: longhorn

$ kubectl apply -f mk8s-iris.yml
```

> The fsGroup is specified because the owner of the mounted Longhorn-derived volume was set to root.  Without it, a protection error will occur when creating the database.
> ```
> irisowner@data-0:~$ ls / -l
> drwxr-xr-x   3 root      root         4096 May 18 15:40 vol-data
> ```
You can delete Longhorn with the following command if you no longer need it.
```
$ kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
```

In case Longhorn was not removed cleanly the last time you used it, you may get the following error:
```
$ kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
  ・
  ・
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml": serviceaccounts "longhorn-service-account" is forbidden: unable to create new content in namespace longhorn-system because it is being terminated
Error from server (Forbid
```

Apparently you can remove it by using longhorn-manager found [here](https://github.com/longhorn/longhorn-manager). I personally stumbled upon this, so I hope you find it helpful.

```
$ git clone https://github.com/longhorn/longhorn-manager.git
$ cd longhorn-manager
$ make
$ kubectl create -f deploy/uninstall/uninstall.yaml
podsecuritypolicy.policy/longhorn-uninstall-psp created
serviceaccount/longhorn-uninstall-service-account created
clusterrole.rbac.authorization.k8s.io/longhorn-uninstall-role created
clusterrolebinding.rbac.authorization.k8s.io/longhorn-uninstall-bind created
job.batch/longhorn-uninstall created
$ kubectl get job/longhorn-uninstall -w
NAME                 COMPLETIONS   DURATION   AGE
longhorn-uninstall   0/1           12s        14s
longhorn-uninstall   1/1           24s        26s
^C
$ kubectl delete -Rf deploy/install
$ kubectl delete -f deploy/uninstall/uninstall.yaml
```

## InterSystems Kubernetes Operator

It also works with IKO on microk8s, however, it's not a feature for Community, thus I decided not to introduce it here.