Highly available IRIS deployment on Kubernetes without mirroring
Tags: aws, tutorial, azure, kubernetes
Published: 2023-03-18 17:34:30

In this article, we’ll build a highly available IRIS configuration using Kubernetes Deployments with distributed persistent storage instead of the “traditional” IRIS mirror pair. This deployment would be able to tolerate infrastructure-related failures, such as node, storage and Availability Zone failures. The described approach greatly reduces the complexity of the deployment at the expense of slightly extended RTO.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5byy0e5f7d0kifkgw7l9.png)

</p><span style="font-size:9pt"><span style="font-family:Calibri,sans-serif"><span style="color:#44546a"><span style="font-style:italic">Figure 1 - Traditional Mirroring vs Kubernetes with Distributed Storage</span></span></span></span></p>

<h1 style="margin-top:16px"><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">All the source code for this article is available at&nbsp;<a href="https://github.com/antonum/ha-iris-k8s" style="color:#0563c1; text-decoration:underline">https://github.com/antonum/ha-iris-k8s</a> </span></span><br><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">TL;DR</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="background:white"><span style="font-family:&quot;Times New Roman&quot;,serif"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">Assuming you have a running 3 node cluster and have some familiarity with Kubernetes – go right ahead:</span></span></span></span></span><span style="background-color: transparent; color: rgb(17, 17, 17); font-size: 12.6px; white-space: pre-wrap;"> </span></p>
<pre><code>kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
kubectl apply -f https://github.com/antonum/ha-iris-k8s/raw/main/tldr.yaml</code></pre>
<p style="margin-bottom:16px"><span style="font-size:12pt"><span style="background:white"><span style="font-family:&quot;Times New Roman&quot;,serif"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">If you are not sure what the two lines above are about or don’t have the system to execute these on – skip to the “High Availability Requirements” section. We’ll explain things in the details as we go. </span></span></span></span></span></p>

<p><span style="font-size:12pt"><span style="background:white"><span style="font-family:&quot;Times New Roman&quot;,serif"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">The first line installs Longhorn - open-source distributed Kubernetes storage. The second one installs InterSystems IRIS deployment, using Longhorn-based volume for Durable SYS. </span></span></span></span></span></p>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="background:white"><span style="font-family:&quot;Times New Roman&quot;,serif"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">Wait for all the pods to come up to the running state. </span></span><code style="font-family:&quot;Courier New&quot;"><span style="border:none windowtext 1.0pt; font-size:10.0pt; padding:0in"><span style="font-family:Consolas"><span style="color:#24292e">kubectl get pods -A </span></span></span></code></span></span></span></p>
<p><span style="font-size:10pt"><span style="font-family:&quot;Courier New&quot;"><span style="font-size:12.0pt"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">You now should be able to access the IRIS management portal at&nbsp;</span></span></span><span class="MsoHyperlink" style="color:#0563c1"><span style="text-decoration:underline"><span style="font-size:12.0pt"><span style="font-family:&quot;Segoe UI&quot;,sans-serif">http://&lt;IRIS Service Public IP&gt;:52773/csp/sys/%25CSP.Portal.Home.zen</span></span></span></span><span style="font-size:12.0pt"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">&nbsp; (default password is 'SYS') and IRIS command line via: </span></span></span></span></span></p>
<pre><code>kubectl exec -it iris-podName-xxxx -- iris session iris</code></pre>
<p><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Simulate the Failure</span></span></span></span></p>
<p style="margin-bottom:16px"><span style="font-size:12pt"><span style="background:white"><span style="font-family:&quot;Times New Roman&quot;,serif"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">Now start messing around. But before you do it, try to add some data into the database and make sure it's there when IRIS is back online.</span></span></span></span></span></p>

<pre><code>kubectl exec -it iris-6d8896d584-8lzn5 -- iris session iris
USER&gt;set ^k8stest($i(^k8stest))=$zdt($h)_" running on "_$system.INetInfo.LocalHostName()
USER&gt;zw ^k8stest
^k8stest=1
^k8stest(1)="01/14/2021 14:13:19 running on iris-6d8896d584-8lzn5"</code></pre>
<p style="margin-bottom:16px"><span style="font-size:12pt"><span style="background:white"><span style="font-family:&quot;Times New Roman&quot;,serif"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:#24292e">Our "chaos engineering" starts here:</span></span></span></span></span></p>

<pre><code># Stop IRIS - Container will be restarted automatically
kubectl exec -it iris-6d8896d584-8lzn5 -- iris stop iris quietly
&nbsp;
# Delete the pod - Pod will be recreated
kubectl delete pod iris-6d8896d584-8lzn5
&nbsp;
# "Force drain" the node, serving the iris pod - Pod would be recreated on another node
kubectl drain aks-agentpool-29845772-vmss000001 --delete-local-data --ignore-daemonsets --force
&nbsp;
# Delete the node - Pod would be recreated on another node
# well... you can't really do it with kubectl. Find that instance or VM and KILL it.
# if you have access to the machine - turn off the power or disconnect the network cable. Seriously!</code></pre>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">High Availability Requirements</span></span></span></span></h1>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">We are building a system that can tolerate a failure of the following:</span></span></p>
<ul>
<li style="margin-left:8px"><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">IRIS instance within container/VM. IRIS – level failure.</span></span></li>
<li style="margin-left:8px"><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Pod/Container failure. </span></span></li>
<li style="margin-left:8px"><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Temporary unavailability of the individual cluster node. A good example would be the Availability Zone temporary goes off-line.</span></span></li>
<li style="margin-left:8px"><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Permanent failure of individual cluster node or disk.</span></span></li>
</ul>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Basically, the scenarios we just tried in the “Simulate the failure” section.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">If any of these failures occur, the system should get online without any human involvement and without data loss. Technically there are limits on what data persistence guarantees. IRIS itself can provide based on the Journal Cycle and transaction usage within an application: <a href="https://docs.intersystems.com/irisforhealthlatest/csp/docbook/Doc.View.cls?KEY=GCDI_journal#GCDI_journal_writecycle" style="color:#0563c1; text-decoration:underline"><span style="background:#fbfbfb"><span style="font-family:&quot;Segoe UI&quot;,sans-serif">https://docs.intersystems.com/irisforhealthlatest/csp/docbook/Doc.View.cls?KEY=GCDI_journal#GCDI_journal_writecycle</span></span></a><span style="background:#fbfbfb"><span style="font-family:&quot;Segoe UI&quot;,sans-serif"><span style="color:black"> In any case, we are talking under two seconds for RPO (Recovery Point Objective).</span></span></span></span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Other components of the system (Kubernetes API Service, etcd database, LoadBalancer service, DNS and others) are outside of the scope and typically managed by the Managed Kubernetes Service such as Azure AKS or AWS EKS so we assume that they are highly available already.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Another way of looking at it – we are responsible for handling individual compute and storage component failures and assuming that the rest is taken care of by the infrastructure/cloud provider.</span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Architecture</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">When it comes to high availability for InterSystems IRIS, the traditional recommendation is to use mirroring. With mirroring you have two always-on IRIS instances synchronously replicating data. Each node maintains a full copy of the database and if the Primary node goes down, users reconnect to the Backup node. Essentially, with the mirroring approach, IRIS is responsible for the redundancy of both compute and storage.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">With mirrors deployed in different availability zones, mirroring provides required redundancy for both compute and storage failure and allows for the excellent RTO (Recovery Time Objective or the time it takes for a system to get back online after a failure) of just a few seconds. You can find the deployment template for Mirrored IRIS on AWS Cloud here: <a href="https://community.intersystems.com/post/intersystems-iris-deployment%C2%A0guide-aws%C2%A0using-cloudformation-template" style="color:#0563c1; text-decoration:underline">https://community.intersystems.com/post/intersystems-iris-deployment%C2%A0guide-aws%C2%A0using-cloudformation-template</a> </span></span></p>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">The less pretty side of mirroring is the complexity of setting it up, performing backup/restore procedures and the lack of replication for security settings and local non-database files.</span></span></p>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Container orchestrators such as Kubernetes (wait, it’s 2021… are there any other left?!) provide compute redundancy via Deployment objects<a>,&nbsp;</a>automatically restarting the failed IRIS Pod/Container in case of failure. That’s why you see only one IRIS node running on the Kubernetes architecture diagram. Instead of keeping a second IRIS node always running we outsource the compute availability to Kubernetes. Kubernetes will make sure that the IRIS pod be recreated in case the original pod fails for whatever reason.</span></span></p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zueko83c1zgkz6rxjvst.png)

<p class="MsoCaption" style="margin-bottom:13px"><span style="font-size:9pt"><span style="font-family:Calibri,sans-serif"><span style="color:#44546a"><span style="font-style:italic">Figure 2 Failover Scenario</span></span></span></span></p>

<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">So far so good… If IRIS node fails, Kubernetes just creates a new one. Depending on your cluster it takes anywhere between 10 and 90 seconds to get IRIS back online after the compute failure. It is a step down compared with just a couple of seconds for mirroring, but if it’s something you can tolerate in the unlikely event of the outage, the reward is the greatly reduced complexity. No mirroring to configure. No security setting and file replication to worry about. </span></span></p>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Frankly, if you login inside the container, running IRIS in Kubernetes, you’ll not even notice that you are running inside the highly available environment. Everything looks and feels just like a single instance IRIS deployment.</span></span></p>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Wait, what about storage? We are dealing with a database nevertheless … Whatever failover scenario we can imagine, our system should take care of the data persistence too. Mirroring relies on the compute, local to the IRIS node. If the node dies or just becomes temporarily unavailable – so does the storage for that node. That’s why in mirroring configuration IRIS takes care of replicating databases on the IRIS level.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">We need storage that can not only preserve the state of the database upon container restart but also can provide redundancy for the event like node or entire segment of the network (Availability Zone) going down. Just a few years ago there was no easy answer to this. As you can guess from the diagram above – we have such an answer now. It is called <b>distributed container storage</b>. </span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Distributed storage abstracts underlying host volumes and presents them as one joint storage available to every node of the k8s cluster. We use Longhorn <a href="https://longhorn.io" style="color:#0563c1; text-decoration:underline">https://longhorn.io</a> in this article; it’s free, open-source and fairly easy to install. But you can also take a look at others, such as OpenEBS, Portworx and StorageOS that would provide the same functionality. Rook Ceph is another CNCF Incubating project to consider. On the high end of the spectrum – there are enterprise-grade storage solutions such as NetApp, PureStorage and others.</span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Step by step guide</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">In TL;DR section we just installed the whole thing in one shot. Appendix B would guide you through step- by step installation and validation procedures.</span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Kubernetes Storage</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Let’s step back for a second and talk about containers and storage in general and how IRIS fits into the picture.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">By default all data inside the container is ephemeral. When the container dies, data disappears. In Docker, you can use the concept of volumes. Essentially it allows you to expose the directory on the host OS to the container.</span></span></p>
<pre><code>docker run --detach
&nbsp; --publish 52773:52773
&nbsp; --volume /data/dur:/dur
&nbsp; --env ISC_DATA_DIRECTORY=/dur/iconfig
&nbsp; --name iris21 --init intersystems/iris:2020.3.0.221.0</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">In the example above we are starting the IRIS container and making the host-local ‘/data/dur’ directory accessible to the container at the ‘/dur’ mount point. So, if the container is storing anything inside this directory, it would be preserved and available to use on the next container start.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">On the IRIS side of things, we can instruct IRIS to store all the data that needs to survive container restart in the specific directory by specifying ISC_DATA_DIRECTORY. Durable SYS is the name of the IRIS feature you might need to look for in the documentation <a href="https://docs.intersystems.com/irisforhealthlatest/csp/docbook/Doc.View.cls?KEY=ADOCK#ADOCK_iris_durable_running" style="color:#0563c1; text-decoration:underline"><span style="background:#fbfbfb"><span style="font-family:&quot;Segoe UI&quot;,sans-serif">https://docs.intersystems.com/irisforhealthlatest/csp/docbook/Doc.View.cls?KEY=ADOCK#ADOCK_iris_durable_running</span></span></a> </span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">In Kubernetes the syntax is different, but the concepts are the same.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Here is the basic Kubernetes Deployment for IRIS.</span></span></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
&nbsp; name: iris
spec:
&nbsp; selector:
&nbsp;&nbsp;&nbsp; matchLabels:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; app: iris
&nbsp; strategy:
&nbsp;&nbsp;&nbsp; type: Recreate
&nbsp; replicas: 1
&nbsp; template:
&nbsp;&nbsp;&nbsp; metadata:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; labels:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; app: iris
&nbsp;&nbsp;&nbsp; spec:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; containers:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - image: store/intersystems/iris-community:2020.4.0.524.0
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: iris
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; env:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: ISC_DATA_DIRECTORY
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; value: /external/iris
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ports:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - containerPort: 52773
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: smp-http
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;volumeMounts:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: iris-external-sys
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mountPath: /external
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; volumes:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: iris-external-sys
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; persistentVolumeClaim:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; claimName: iris-pvc</code></pre>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">In the deployment specification above, ‘volumes’ part lists storage volumes. They can be available outside of the container, via persistentVolumeClaim such as ‘iris-pvc’. volumeMounts expose this volume inside the container. ‘iris-external-sys’ is the identifier that ties volume mount to the specific volume. In reality, we might have multiple volumes and this name is used just to distinguish one from another. You can call it ‘steve’ if you want.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Already familiar environment variable ISC_DATA_DIRECTORY directs IRIS to use a specific mount point to store all the data that needs to survive container restart.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Now let’s take a look at the Persistent Volume Claim iris-pvc.</span></span></p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
&nbsp; name: iris-pvc
spec:
&nbsp; storageClassName: longhorn
&nbsp; accessModes:
&nbsp;&nbsp;&nbsp; - ReadWriteOnce
&nbsp; resources:
&nbsp;&nbsp;&nbsp; requests:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; storage: 10Gi</code></pre>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Fairly straightforward. Requesting 10 gigabytes, mountable as Read/Write on one node only, using storage class of ‘longhorn’. </span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">That <span style="font-size:8.0pt"><span style="font-family:&quot;Andale Mono&quot;">storageClassName: longhorn</span></span> is actually critical here.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Let’s look at what storage classes are available on my AKS cluster:</span></span></p>
<pre><code>kubectl get StorageClass
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PROVISIONER&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RECLAIMPOLICY&nbsp;&nbsp; VOLUMEBINDINGMODE&nbsp;&nbsp; ALLOWVOLUMEEXPANSION&nbsp;&nbsp; AGE
azurefile&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; kubernetes.io/azure-file&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Delete&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Immediate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; true&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10d
azurefile-premium&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; kubernetes.io/azure-file&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Delete&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Immediate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; true&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10d
default (default)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; kubernetes.io/azure-disk&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Delete&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Immediate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; true&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10d
longhorn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; driver.longhorn.io&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Delete&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Immediate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; true&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10d
managed-premium&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; kubernetes.io/azure-disk&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Delete&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Immediate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; true&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10d</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">There are few storage classes from Azure, installed by default and one from Longhorn that we installed as part of the very first command:</span></span></p>
<pre><code>kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">If you comment out <span style="font-size:8.0pt"><span style="font-family:&quot;Andale Mono&quot;">#storageClassName: longhorn</span></span> in the Persistent Volume Claim definition, it will use storage class, currently marked as “default” which is a regular Azure Disk.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">To illustrate why we need Distributed storage let’s repeat the “chaos engineering” experiments we described at the beginning of the article without longhorn storage. The first two scenarios (stop IRIS and delete the Pod) would successfully complete and systems would recover to the operational state. Attempting to either drain or kill the node would bring the system into a failed state.</span></span></p>
<pre><code>#forcefully drain the node
kubectl drain aks-agentpool-71521505-vmss000001 --delete-local-data --ignore-daemonsets

kubectl describe pods
...
&nbsp; Type&nbsp;&nbsp;&nbsp;&nbsp; Reason&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Age&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; From&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Message
&nbsp; ----&nbsp;&nbsp;&nbsp;&nbsp; ------&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ----&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ----&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;-------
&nbsp; Warning&nbsp; FailedScheduling&nbsp; 57s (x9 over 2m41s)&nbsp; default-scheduler&nbsp; 0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) had volume node affinity conflict.</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Essentially, Kubernetes would try to restart the IRIS pod on the cluster, but the node where it was originally started is not available and the other two nodes have “volume node affinity conflict”. With this storage type, the volume is available only on the node it was originally created since it is basically tied to the disk available on the node host.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">With longhorn as a storage class, both “force drain” and “node kill” experiments succeed, and the IRIS pod is back into operation shortly. To achieve it Longhorn takes control over the available storage on the 3 nodes of the cluster and replicates the data across all three nodes. Longhorn promptly repairs cluster storage if one of the nodes becomes permanently unavailable. In our “node kill” scenario, the IRIS pod is restarted on another node right away using two remaining volume replicas. &nbsp;Then, AKS provisions a new node to replace the lost one and as soon as it is ready, Longhorn kicks in and rebuilds required data on the new node. Everything is automatic, without your involvement.</span></span></p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dikx0fj586tamvusdwpx.png)

<p class="MsoCaption" style="margin-bottom:13px"><span style="font-size:9pt"><span style="font-family:Calibri,sans-serif"><span style="color:#44546a"><span style="font-style:italic">Figure 3 Longhorn rebuilding volume replica on the replaced node</span></span></span></span></p>

<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">More about k8s deployment</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Let’s take a look at some other aspects of our deployment:</span></span></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
&nbsp; name: iris
spec:
&nbsp; selector:
&nbsp;&nbsp;&nbsp; matchLabels:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; app: iris
&nbsp; strategy:
&nbsp;&nbsp;&nbsp; type: Recreate
&nbsp; replicas: 1
&nbsp; template:
&nbsp;&nbsp;&nbsp; metadata:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; labels:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; app: iris
&nbsp;&nbsp;&nbsp; spec:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; containers:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - image: store/intersystems/iris-community:2020.4.0.524.0
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: iris
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; env:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: ISC_DATA_DIRECTORY
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; value: /external/iris
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: ISC_CPF_MERGE_FILE
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; value: /external/merge/merge.cpf
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ports:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - containerPort: 52773
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: smp-http
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; volumeMounts:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: iris-external-sys
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mountPath: /external
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: cpf-merge
&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mountPath: /external/merge
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; livenessProbe:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; initialDelaySeconds: 25
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; periodSeconds: 10
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; exec:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; command:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - /bin/sh
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - -c
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - "iris qlist iris | grep running"
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; volumes:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: iris-external-sys
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; persistentVolumeClaim:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; claimName: iris-pvc
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: cpf-merge
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; configMap:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: iris-cpf-merge</code></pre>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>strategy: Recreate, replicas: 1</b> tells Kubernetes that at any given time it should maintain one and exactly one instance of IRIS pod running. This is what takes care of our “delete pod” scenario.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>livenessProbe</b> section makes sure that IRIS is always up inside the container and handles “IRIS is down” scenario. initialDelaySeconds allows for some grace period for IRIS to start. You might want to increase it if IRIS is taking a considerable time to start your deployment.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>CPF MERGE</b> feature of IRIS allows you to modify the content of the configuration file iris.cpf upon container start. See <a href="https://docs.intersystems.com/irisforhealthlatest/csp/docbook/DocBook.UI.Page.cls?KEY=RACS_cpf#RACS_cpf_edit_merge" style="color:#0563c1; text-decoration:underline">https://docs.intersystems.com/irisforhealthlatest/csp/docbook/DocBook.UI.Page.cls?KEY=RACS_cpf#RACS_cpf_edit_merge</a> for relevant documentation. In this example I’m using Kubernetes Config Map to manage the content of the merge file: <a href="https://github.com/antonum/ha-iris-k8s/blob/main/iris-cpf-merge.yaml" style="color:#0563c1; text-decoration:underline">https://github.com/antonum/ha-iris-k8s/blob/main/iris-cpf-merge.yaml</a> Here we adjust global buffers and gmheap values, used by IRIS instance, but everything you can find in iris.cpf file is a fair game. You can even change the default IRIS password using `PasswordHash` field in the CPF Merge file. Read more at: <a href="https://docs.intersystems.com/irisforhealthlatest/csp/docbook/Doc.View.cls?KEY=ADOCK#ADOCK_iris_images_password_auth" style="color:#0563c1; text-decoration:underline">https://docs.intersystems.com/irisforhealthlatest/csp/docbook/Doc.View.cls?KEY=ADOCK#ADOCK_iris_images_password_auth</a> </span></span></p>
<p>&nbsp;</p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Besides Persistent Volume Claim <a href="https://github.com/antonum/ha-iris-k8s/blob/main/iris-pvc.yaml" style="color:#0563c1; text-decoration:underline">https://github.com/antonum/ha-iris-k8s/blob/main/iris-pvc.yaml</a> deployment <a href="https://github.com/antonum/ha-iris-k8s/blob/main/iris-deployment.yaml" style="color:#0563c1; text-decoration:underline">https://github.com/antonum/ha-iris-k8s/blob/main/iris-deployment.yaml</a> and ConfigMap with CPF Merge content <a href="https://github.com/antonum/ha-iris-k8s/blob/main/iris-cpf-merge.yaml" style="color:#0563c1; text-decoration:underline">https://github.com/antonum/ha-iris-k8s/blob/main/iris-cpf-merge.yaml</a> our deployment needs a service that exposes IRIS deployment to the public internet: <a href="https://github.com/antonum/ha-iris-k8s/blob/main/iris-svc.yaml" style="color:#0563c1; text-decoration:underline">https://github.com/antonum/ha-iris-k8s/blob/main/iris-svc.yaml</a> </span></span></p>
<pre><code>kubectl get svc
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; TYPE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CLUSTER-IP&nbsp;&nbsp;&nbsp; EXTERNAL-IP&nbsp;&nbsp;&nbsp;&nbsp; PORT(S)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AGE
iris-svc&nbsp;&nbsp;&nbsp;&nbsp; LoadBalancer&nbsp;&nbsp; 10.0.18.169&nbsp;&nbsp; 40.88.123.45&nbsp;&nbsp; 52773:31589/TCP&nbsp;&nbsp; 3d1h
kubernetes&nbsp;&nbsp; ClusterIP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10.0.0.1&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&lt;none&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 443/TCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10d</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">External IP of the iris-svc can be used to access the IRIS management portal via&nbsp;<a href="http://40.88.123.45:52773/csp/sys/%25CSP.Portal.Home.zen" style="color:#0563c1; text-decoration:underline">http://40.88.123.45:52773/csp/sys/%25CSP.Portal.Home.zen</a>. The default password is 'SYS'. </span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Backup/Restore and Storage Scaling</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Longhorn provides web-based UI for configuring and managing volumes.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Identify the pod, running longhorn-ui component and establish port forwarding with kubectl:</span></span></p>
<pre><code>kubectl -n longhorn-system get pods
# note the longhorn-ui pod id.

kubectl port-forward longhorn-ui-df95bdf85-gpnjv 9000:8000 -n longhorn-system</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Longhorn UI will become available at <a href="http://localhost:9000" style="color:#0563c1; text-decoration:underline">http://localhost:9000</a></span></span></p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mxz4eq09wrfmeqwsnw08.png)

<p class="MsoCaption" style="margin-bottom:13px"><span style="font-size:9pt"><span style="font-family:Calibri,sans-serif"><span style="color:#44546a"><span style="font-style:italic">Figure 4 Longhorn UI</span></span></span></span></p>

<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Besides high availability, most of the Kubernetes container storage solutions provide convenient options for backup, snapshots and restore. Details are implementation-specific, but the common convention is that backup is associated with the VolumeSnapshot. It is so for Longhorn. Depending on your Kubernetes version and provider you might also need to install volume snapshotter <a href="https://github.com/kubernetes-csi/external-snapshotter" style="color:#0563c1; text-decoration:underline">https://github.com/kubernetes-csi/external-snapshotter</a> </span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">`iris-volume-snapshot.yaml` is an example of such a volume snapshot. Before using it, you need to configure backups to either the S3 bucket or NFS volume in Longhorn. https://longhorn.io/docs/1.0.1/snapshots-and-backups/backup-and-restore/set-backup-target/ </span></span></p>
<pre><code># Take crash-consistent backup of the iris volume
kubectl apply -f iris-volume-snapshot.yaml</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">For IRIS it is recommended that you execute External Freeze before taking the backup/snapshot and Thaw after. See details here: <a href="https://docs.intersystems.com/irisforhealthlatest/csp/documatic/%25CSP.Documatic.cls?LIBRARY=%25SYS&amp;CLASSNAME=Backup.General#ExternalFreeze" style="color:#0563c1; text-decoration:underline">https://docs.intersystems.com/irisforhealthlatest/csp/documatic/%25CSP.Documatic.cls?LIBRARY=%25SYS&amp;CLASSNAME=Backup.General#ExternalFreeze</a> &nbsp;</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">To increase the size of the IRIS volume - adjust storage request in persistent volume claim (file `iris-pvc.yaml`), used by IRIS.</span></span></p>
<pre><code>...
&nbsp; resources:
&nbsp;&nbsp;&nbsp; requests:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; storage: 10Gi #change this value to required</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Then, re-apply the pvc specification. Longhorn cannot actually apply this change while the volume is connected to the running Pod. Temporarily change replicas count to zero in the deployment so volume size can be increased.</span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">High Availability – Overview</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">At the beginning of the article, we set some criteria for High Availability. Here is how we achieve it with this architecture:</span></span></p>
<table class="MsoTableGrid" style="border-collapse:collapse; border:none">
<tbody>
<tr>
<td style="border-bottom:1px solid black; width:294px; padding:0in 7px 0in 7px; border-top:1px solid black; border-right:1px solid black; border-left:1px solid black" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>Failure Domain</b></span></span></p></td>
<td style="border-bottom:1px solid black; width:330px; padding:0in 7px 0in 7px; border-top:1px solid black; border-right:1px solid black; border-left:none" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>Automatically mitigated by</b></span></span></p></td>
</tr>
<tr>
<td style="border-bottom:1px solid black; width:294px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:1px solid black" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">IRIS instance within container/VM. IRIS – level failure.</span></span></p></td>
<td style="border-bottom:1px solid black; width:330px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:none" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Deployment Liveness probe restarts container in case IRIS is down</span></span></p></td>
</tr>
<tr>
<td style="border-bottom:1px solid black; width:294px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:1px solid black" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Pod/Container failure. </span></span></p></td>
<td style="border-bottom:1px solid black; width:330px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:none" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Deployment recreates Pod</span></span></p></td>
</tr>
<tr>
<td style="border-bottom:1px solid black; width:294px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:1px solid black" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Temporary unavailability of the individual cluster node. A good example would be Availability Zone going off-line.</span></span></p></td>
<td style="border-bottom:1px solid black; width:330px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:none" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Deployment recreates pod on another node. Longhorn makes data available on another node.</span></span></p></td>
</tr>
<tr>
<td style="border-bottom:1px solid black; width:294px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:1px solid black" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Permanent failure of individual cluster node or disk.</span></span></p></td>
<td style="border-bottom:1px solid black; width:330px; padding:0in 7px 0in 7px; border-top:none; border-right:1px solid black; border-left:none" valign="top">
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Same as above + k8s cluster autoscaler replaces a damaged node with a new one. Longhorn rebuilds data on the new node.</span></span></p></td>
</tr>
</tbody></table>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Zombies and other things to consider</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">If you are familiar with running IRIS in the Docker containers, you might have used the `--init` flag.</span></span></p>
<pre><code>docker run --rm -p 52773:52773 --init store/intersystems/iris-community:2020.4.0.524.0</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">The goal of this flag is to prevent the formation of the "zombie processes". In Kubernetes, you can either use ‘shareProcessNamespace: true’ (security considerations apply) or in your own containers utilize `tini`. Example Dockerfile with tini:</span></span></p>
<pre><code>FROM iris-community:2020.4.0.524.0
...
# Add Tini
USER root
ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
USER irisowner
ENTRYPOINT ["/tini", "--", "/iris-main"]</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Starting 2021, all InterSystems provided container images would include tini by default.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">You can further decrease the failover time for “force drain node/kill node” scenarios by adjusting few parameters:</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Longhorn Pod Deletion Policy <a href="https://longhorn.io/docs/1.1.0/references/settings/#pod-deletion-policy-when-node-is-down" style="color:#0563c1; text-decoration:underline"><span style="font-size:10.5pt"><span style="background:white"><span style="font-family:&quot;Segoe UI&quot;,sans-serif">https://longhorn.io/docs/1.1.0/references/settings/#pod-deletion-policy-when-node-is-down</span></span></span></a> and kubernetes taint-based eviction: <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions" style="color:#0563c1; text-decoration:underline"><span style="font-size:10.5pt"><span style="background:white"><span style="font-family:&quot;Segoe UI&quot;,sans-serif">https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions</span></span></span></a></span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Disclaimer</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">As the InterSystems employee, I kinda have to put this in here: <i>Longhorn is used in this article as an example of distributed Kubernetes Block Storage. InterSystems does not validate or issue an official support statement for individual storage solutions or products. You need to test and validate if any specific storage solution fits your needs.</i></span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Distributed storage might also have substantially different performance characteristics, comparing to node-local storage. Especially for write operations, where data must be written to multiple locations before it is considered to be in the persisted state. Make sure to test your workloads and understand the specific behaviour and options your CSI driver offers..</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><a>Basically, InterSystems does not validate and/or endorse specific storage solutions like Longhorn in the same way as we don’t validate individual HDD brands or server hardware manufacturers</a>. I personally found Longhorn easy to work with and their development team extremely responsive and helpful at the project’s GitHub page.&nbsp;<a href="https://github.com/longhorn/longhorn" target="_blank">https://github.com/longhorn/longhorn</a>&nbsp;</span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Conclusion</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Kubernetes ecosystem evolved significantly in the past few years and with the use of distributed block storage solutions, you now can build a Highly Available configuration that can sustain IRIS instance, cluster node and even Availability Zone failure.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">You can outsource compute and storage high availability to Kubernetes components, resulting in a significantly simpler system to configure and maintain, comparing to the traditional IRIS mirroring. At the same time, this configuration might not provide the same RTO and storage – level performance as mirrored configuration.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">In this article, we build a highly available IRIS configuration using Azure AKS as a managed Kubernetes and Longhorn distributed storage system. You can explore multiple alternatives such as AWS EKS, Google Kubernetes Engine for managed K8s, StorageOS, Portworx and OpenEBS as distributed container storage or even enterprise-level storage solutions such as NetApp, PureStorage, Dell EMC and others. </span></span></p>
<h1 style="margin-top:16px"><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Appendix A. Creating Kubernetes Cluster in the cloud</span></span></span></span></h1>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Managed Kubernetes service from one of the public cloud providers is an easy way to create k8s cluster required for this setup. Azure’s AKS default configuration is ready out of the box to be used for the deployment described in this article.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Create a new AKS cluster with 3 nodes. Leave everything else default.</span></span></p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/st8mjend1cmnf9za94vh.png)

<p class="MsoCaption" style="margin-bottom:13px"><span style="font-size:9pt"><span style="font-family:Calibri,sans-serif"><span style="color:#44546a"><span style="font-style:italic">Figure 5 Create AKS cluster</span></span></span></span></p>

<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Install kubectl on your computer locally: <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/" style="color:#0563c1; text-decoration:underline">https://kubernetes.io/docs/tasks/tools/install-kubectl/</a> </span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Register your AKS cluster with local kubectl</span></span></p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gm35a4d2omr0ytm4ji8o.png)

<p><span style="font-size:9pt"><span style="font-family:Calibri,sans-serif"><span style="color:#44546a"><span style="font-style:italic">Figure 6 Register AKS cluster with kubectl</span></span></span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">After that, you can get right back to the beginning of the article and install longhorn and IRIS deployment.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Installation on AWS EKS is a little bit more complicated. You need to make sure every instance in your node group has open-iscsi installed.</span></span></p>
<pre><code>sudo yum install iscsi-initiator-utils -y</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Installing Longhorn on GKE requires extra step, described here: <a href="https://longhorn.io/docs/1.0.1/advanced-resources/os-distro-specific/csi-on-gke/" style="color:#0563c1; text-decoration:underline">https://longhorn.io/docs/1.0.1/advanced-resources/os-distro-specific/csi-on-gke/</a> </span></span></p>
<p><span style="font-size:16pt"><span style="font-family:&quot;Calibri Light&quot;,sans-serif"><span style="color:#2f5496"><span style="font-weight:normal">Appendix B. Step by step installation</span></span></span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>Step 1 – Kubernetes Cluster and kubectl</b></span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">You need 3 nodes k8s cluster. Appendix A describes how to get one on Azure.</span></span></p>
<pre><code>$ kubectl get nodes
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; STATUS&nbsp;&nbsp; ROLES&nbsp;&nbsp; AGE&nbsp;&nbsp; VERSION
aks-agentpool-29845772-vmss000000&nbsp;&nbsp; Ready&nbsp;&nbsp;&nbsp; agent&nbsp;&nbsp; 10d&nbsp;&nbsp; v1.18.10
aks-agentpool-29845772-vmss000001&nbsp;&nbsp; Ready&nbsp;&nbsp;&nbsp; agent&nbsp;&nbsp; 10d&nbsp;&nbsp; v1.18.10
aks-agentpool-29845772-vmss000002&nbsp;&nbsp; Ready&nbsp;&nbsp; &nbsp;agent&nbsp;&nbsp; 10d&nbsp;&nbsp; v1.18.10</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>Step 2 – Install Longhorn</b></span></span></p>
<pre><code>kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Make sure all the pods in the ‘longhorn-system’ namespace are in the running state. It might take few minutes.</span></span></p>
<pre><code>$ kubectl get pods -n longhorn-system
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; READY&nbsp;&nbsp; STATUS&nbsp;&nbsp;&nbsp; RESTARTS&nbsp;&nbsp; AGE
csi-attacher-74db7cf6d9-jgdxq&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;1/1&nbsp;&nbsp;&nbsp;&nbsp; Running&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10d
csi-attacher-74db7cf6d9-l99fs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1/1&nbsp;&nbsp;&nbsp;&nbsp; Running&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 11d
...
longhorn-manager-flljf&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1/1&nbsp;&nbsp;&nbsp;&nbsp; Running&nbsp;&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 11d
longhorn-manager-x76n2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1/1&nbsp;&nbsp;&nbsp;&nbsp; Running&nbsp;&nbsp; 1&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11d
longhorn-ui-df95bdf85-gpnjv&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1/1&nbsp;&nbsp;&nbsp;&nbsp; Running&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 11d</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Refer to the Longhorn installation guide for details and troubleshooting <a href="https://longhorn.io/docs/1.1.0/deploy/install/install-with-kubectl" style="color:#0563c1; text-decoration:underline">https://longhorn.io/docs/1.1.0/deploy/install/install-with-kubectl</a> </span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>Step 3 – Clone the GitHub repo</b></span></span></p>
<pre><code>$ git clone https://github.com/antonum/ha-iris-k8s.git
$ cd ha-iris-k8s
$ ls
LICENSE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; iris-deployment.yaml&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; iris-volume-snapshot.yaml
README.md &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;iris-pvc.yaml&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; longhorn-aws-secret.yaml
iris-cpf-merge.yaml&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; iris-svc.yaml&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tldr.yaml</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>Step 4 – deploy and validate components one by one</b></span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">tldr.yaml file contains all the components needed for the deployment in one bundle. Here we’ll install them one by one and validate the setup of every one of them individually.</span></span></p>
<pre><code># If you have previously applied tldr.yaml - delete it.
$ kubectl delete -f https://github.com/antonum/ha-iris-k8s/raw/main/tldr.yaml

# Create Persistent Volume Claim
$ kubectl apply -f iris-pvc.yaml
persistentvolumeclaim/iris-pvc created

$ kubectl get pvc
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; STATUS&nbsp;&nbsp; VOLUME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CAPACITY&nbsp;&nbsp; ACCESS MODES&nbsp;&nbsp; STORAGECLASS&nbsp;&nbsp; AGE
iris-pvc&nbsp;&nbsp; Bound&nbsp;&nbsp;&nbsp; pvc-fbfaf5cf-7a75-4073-862e-09f8fd190e49&nbsp;&nbsp; 10Gi&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RWO&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; longhorn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10s

# Create Config Map
$ kubectl apply -f iris-cpf-merge.yaml

$ kubectl describe cm iris-cpf-merge
Name:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; iris-cpf-merge
Namespace:&nbsp;&nbsp;&nbsp; default
Labels:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;none&gt;
Annotations:&nbsp; &lt;none&gt;

Data
====
merge.cpf:
----
[config]
globals=0,0,800,0,0,0
gmheap=256000
Events:&nbsp; &lt;none&gt;

# create iris deployment
$ &nbsp;kubectl apply -f iris-deployment.yaml
deployment.apps/iris created

$ kubectl get pods&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; READY&nbsp;&nbsp; STATUS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RESTARTS&nbsp;&nbsp; AGE
iris-65dcfd9f97-v2rwn&nbsp;&nbsp; 0/1&nbsp;&nbsp;&nbsp;&nbsp; ContainerCreating&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;11s

# note the pod name. You’ll use it to connect to the pod in the next command

$ kubectl exec -it iris-65dcfd9f97-v2rwn&nbsp;&nbsp; -- bash

irisowner@iris-65dcfd9f97-v2rwn:~$ iris session iris
Node: iris-65dcfd9f97-v2rwn, Instance: IRIS

USER&gt;w $zv
IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2020.4 (Build 524U) Thu Oct 22 2020 13:04:25 EDT
# h&lt;enter&gt; to exit IRIS shell
# exit&lt;enter&gt; to exit pod

# access the logs of the IRIS container
$ kubectl logs iris-65dcfd9f97-v2rwn
...
[INFO] ...started InterSystems IRIS instance IRIS
01/18/21-23:09:11:312 (1173) 0 [Utility.Event] Private webserver started on 52773
01/18/21-23:09:11:312 (1173) 0 [Utility.Event] Processing Shadows section (this system as shadow)
01/18/21-23:09:11:321 (1173) 0 [Utility.Event] Processing Monitor section
01/18/21-23:09:11:381 (1323) 0 [Utility.Event] Starting TASKMGR
01/18/21-23:09:11:392 (1324) 0 [Utility.Event] [SYSTEM MONITOR] System Monitor started in %SYS
01/18/21-23:09:11:399 (1173) 0 [Utility.Event] Shard license: 0
01/18/21-23:09:11:778 (1162) 0 [Database.SparseDBExpansion] Expanding capacity of sparse database /external/iris/mgr/iristemp/ by 10 MB.

# create iris service
$ kubectl apply -f iris-svc.yaml&nbsp;&nbsp;&nbsp;
service/iris-svc created

$ kubectl get svc
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;TYPE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CLUSTER-IP&nbsp;&nbsp;&nbsp;&nbsp; EXTERNAL-IP&nbsp;&nbsp;&nbsp; PORT(S)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AGE
iris-svc&nbsp;&nbsp;&nbsp;&nbsp; LoadBalancer &nbsp;&nbsp;10.0.214.236&nbsp;&nbsp; 20.62.241.89&nbsp;&nbsp; 52773:30128/TCP&nbsp;&nbsp; 15s</code></pre>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif"><b>Step 5 – Access management portal</b></span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">Finally – connect to the management portal of the IRIS, using the external IP of the service: <a href="http://20.62.241.89:52773/csp/sys/%25CSP.Portal.Home.zen" style="color:#0563c1; text-decoration:underline">http://20.62.241.89:52773/csp/sys/%25CSP.Portal.Home.zen</a> username _SYSTEM, Password SYS. You’ll be asked to change it on your first login.</span></span></p>
<p><span style="font-size:12pt"><span style="font-family:Calibri,sans-serif">&nbsp;


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6j6rl4acbhpl87b232pa.png)