YugabyteDB on OKE
Tags: yugabytedb, oraclecloud, kubernetes, database
Published: 2021-09-19 20:02:27

We have documented how to install YugabyteDB on the main cloud providers (<a href="https://docs.yugabyte.com/latest/deploy/public-clouds">https://docs.yugabyte.com/latest/deploy/public-clouds</a>). And when I joined Yugabyte, many friends from the oracle community asked me about the Oracle Cloud (OCI).

In the Oracle Cloud there are 2 ways to run containers:
<ul>
 	<li>Do-It-Yourself with IaaS (Infrastructure as a Service)
Obviously, you can provision some compute instances (and network and storage) running Docker and install Kubernetes (or other orchestration). Oracle provides a Terraform installer for that: <a href="https://github.com/oracle/terraform-kubernetes-installer" target="_blank" rel="noopener">https://github.com/oracle/terraform-kubernetes-installer</a>.</li>
 	<li>Managed with OKE (OCI Container Engine for Kubernetes) to deploy Kubernetes quickly, upgrade easily... The Registry (OCIR) and Kubernetes Engine (OKE) are free. We pay only for the resources (compute instances, storage, load balancers).</li>
</ul>
<h2 style="text-align: left">Developer Services Console</h2>
I’ll use the managed one here, OKE. The Kubernetes service is found in the OCI menu under <i>Developer Services</i>. Then in <i>Containers &amp; Artifacts</i> click on <i>Kubernetes Clusters (OKE).</i>

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2l528zjsarijfdkysdam.png)

You can also access it directly with: <a href="https://cloud.oracle.com/containers/clusters" target="_blank" rel="noopener">https://cloud.oracle.com/containers/clusters</a> and choosing your region. In Oracle Cloud, the price does not depend on the region, so latency with users is the main criteria. This can also be Cloud@Customer where the managed service runs on your data center.

This is where you can see your K8s clusters.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/52h2hto23mzeqb5ni6y1.png)

I’ll choose the <i>Quick Create</i> wizard here.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3rla35emt79konf614ow.png)

The main information to provide is:
<ul>
 	<li>The name of the cluster</li>
 	<li>The version of Kubernetes to run on the master node</li>
</ul>
This setup will create:
<ul>
 	<li>A VCN, with NAT, Internet and Service Gateways</li>
 	<li>3 subnets (one private for the worker nodes, and the other public as I keep the default “public workers” for this demo)</li>
 	<li>2 Load Balancers (for master and tserver), with their security lists</li>
 	<li>The worker nodes and API endpoint</li>
</ul>
I’m using public endpoints here because this is a lab, but obviously you put your database on a private subnet.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ikmvdwc0o03xzfrtxo8.png)

A first Node Pool will be created, with the number of nodes you define here. I define a 4 OCPU (= 4 Intel cores = 8 OS threads) shape per node in this pool and with Flex shapes you can size the CPU and RAM independently. Don’t forget that the database will be stored on each node, with two mounted block volumes. In the <i>Advanced Options</i> you can upload your public ssh keys if you want to access the nodes without kubectl.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u4hv57ie6236o5wqr62i.png)

You can review and create it directly, or “Save as Stack” to create it with Terraform later:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bhemwcz3zkfcvx7wdvnq.png)

You can also look at the support tab which you can access with the new Ring Buoy icon which floats on the right of the screen.

Here are the services being created (with their OCID - the identifier in the Oracle Cloud):

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q0qpik8b3nfy6pnjo8sc.png)

With this I have 3 nodes running, one in each Availability Domain (the OCI equivalent of Availability Zone) which I can see in the “Compute” service, with my other VMs:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5lkqxupv82vhshxa2gso.png)

This is what will be billed. If you want to stop the nodes you can scale down the pool to zero nodes.

Of course there will still be the Block Volumes and Load Balancers as well in the bill when you have all nodes stopped.
<h2 style="text-align: left">Kubeconfig</h2>
Now, the easiest way to install YugabyteDB is to continue with kubectl from the Cloud Shell and have all of the information to set the configuration be visible from the Access Cluster button. This builds the KUBECONFIG.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y6z2j05538ojry995wxi.png)

Here is the command (of course you should not copy/paste the exact below command but copy the detail for your cluster, from <i>Quick Start </i>/<i> Access Cluster</i>):
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>umask 066</b>
dev@cloudshell:~ (uk-london-1)$ <b>oci ce cluster create-kubeconfig --cluster-id ocid1.cluster.oc1.uk-london-1.aaaaaaaadlkettlvl4frdlz6m7j6oiqvcupse4z6qzbdtxw74cbz655xbyfq --file $HOME/.kube/config --region uk-london-1 --token-version 2.0.0  --kube-endpoint PUBLIC_ENDPOINT</b></pre>
I’ve set umask to have it readable by myself only. You can do that afterwards too.
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>chmod 600 $HOME/.kube/config</b></pre>
Check the version:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>kubectl version</b>
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"a84e568eeb56c4e3966314fc2d58374febd12ed7", GitTreeState:"clean", BuildDate:"2021-03-09T14:37:57Z", GoVersion:"go1.13.15 BoringCrypto", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.12", GitCommit:"396aaa6314665118cf108193f0cc2318f7112979", GitTreeState:"clean", BuildDate:"2021-06-30T16:15:03Z", GoVersion:"go1.15.13 BoringCrypto", Compiler:"gc", Platform:"linux/amd64"}</pre>
Helm is already installed:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>helm version</b>
version.BuildInfo{Version:"v3.3.4", GitCommit:"82f031c2db91441a56f7b05640a3dbe211222e81", GitTreeState:"clean", GoVersion:"go1.15.10"}</pre>
<h2 style="text-align: left">Install YugabyteDB</h2>
I’m ready to create a YugabyteDB cluster, from helm charts, which is all documented: <a href="https://docs.yugabyte.com/latest/deploy/kubernetes/single-zone/oss/helm-chart/" target="_blank" rel="noopener">https://docs.yugabyte.com/latest/deploy/kubernetes/single-zone/oss/helm-chart/</a>

Adding the YugabyteDB chart to the repository:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>helm repo add yugabytedb https://charts.yugabyte.com</b>
"yugabytedb" has been added to your repositories</pre>
Update it:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>helm repo update</b>
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "yugabytedb" chart repository
Update Complete. ⎈Happy Helming!⎈</pre>
Checking the YugabyteDB version:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>helm search repo yugabytedb/yugabyte</b>
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
yugabytedb/yugabyte     2.7.1           2.7.1.1-b1      YugabyteDB is the high-performance distributed ...</pre>
Ready to install in its dedicated namespace:
<pre>dev@cloudshell:~ (uk-london-1)$ <b>kubectl create namespace yb-demo</b> 
namespace/yb-demo created

dev@cloudshell:~ (uk-london-1)$ <b>helm install yb-demo yugabytedb/yugabyte --namespace yb-demo --wait --set gflags.tserver.ysql_enable_auth=true,storage.master.storageClass=oci-bv,storage.tserver.storageClass=oci-bv</b>
</pre>
I enabled authentication (default 'yugabyte' user password is 'yugabyte'). Authentication is enabled with theTServer flag 'ysql_enable_auth' set to true.
<h2 style="text-align: left">Note about Storage Class</h2>
Note that on my first attempt, without mentioning StorageClass, I got it failing with something like this because the default is not correct:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>kubectl describe pod --namespace yb-demo</b>
…
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  6m44s  default-scheduler  no nodes available to schedule pods
  Warning  FailedScheduling  6m43s  default-scheduler  no nodes available to schedule pods
  Warning  FailedScheduling  6m39s  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  6m18s  default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
</pre>
The reason for unbound PersistentVolumeClaims was:
<pre class="lang:default decode:true">Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  ProvisioningFailed  4m (x142 over 39m)  persistentvolume-controller  storageclass.storage.k8s.io "standard" not found
</pre>
Let’s check the storageclass:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>kubectl get storageclass</b>
NAME            PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
oci (default)   oracle.com/oci                    Delete          Immediate              false                  3h32m
oci-bv          blockvolume.csi.oraclecloud.com   Delete          WaitForFirstConsumer   false                  3h32m</pre>
This is the reason I mentioned the ‘oci-bv’ (Container Storage Interface to the Block Volumes) storage class with: <code>--set storage.master.storageClass=oci-bv,storage.tserver.storageClass=oci-bv</code>
<h2 style="text-align: left">Checking installation</h2>
The install takes a few minutes and here are the pods up and running:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>kubectl --namespace yb-demo get pods</b>
NAME       READY   STATUS RESTARTS   AGE
yb-master-0 1/1 Running   0      100s
yb-master-1 1/1 Running   0      100s
yb-master-2 1/1 Running   0      100s
yb-tserver-0   1/1 Running   0      100s
yb-tserver-1   1/1 Running   0      100s
yb-tserver-2   1/1 Running   0      100s</pre>
kubectl exec allows you to perform a shell command after '<code>--</code>'. I can check on the filesystems, memory, and cpu for a node in this way:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>kubectl exec --namespace yb-demo -it yb-tserver-0 -- df -Th /mnt/disk0 /mnt/disk1</b>
Defaulting container name to yb-tserver.
Use 'kubectl describe pod/yb-tserver-0 -n yb-demo' to see all of the containers in this pod.
Filesystem Type  Size  Used Avail Use% Mounted on
/dev/sdd   ext4   50G   77M   47G   1% <b>/mnt/disk0</b>
/dev/sde   ext4   50G   59M   47G   1% <b>/mnt/disk1</b>
dev@cloudshell:~ (uk-london-1)$ <b>kubectl exec --namespace yb-demo -it yb-tserver-0 -- free -h</b>
Defaulting container name to yb-tserver.
Use 'kubectl describe pod/yb-tserver-0 -n yb-demo' to see all of the containers in this pod.
total    used    free  shared  buff/cache   available
Mem:        <b>30G</b>    1.1G     23G     36M    6.5G     29G
Swap:        0B      0B      0B
dev@cloudshell:~ (uk-london-1)$ kubectl <b>exec --namespace yb-demo -it yb-tserver-0 -- lscpu</b>
Defaulting container name to yb-tserver.
Use 'kubectl describe pod/yb-tserver-0 -n yb-demo' to see all of the containers in this pod.
Architecture:      x86_64
CPU op-mode(s):    32-bit, 64-bit
Byte Order:        Little Endian
CPU(s):            8
On-line CPU(s) list:   0-7
Thread(s) per core: 2
<b>Core(s) per socket: 4</b>
<b>Socket(s):         1</b>
NUMA node(s):      1
Vendor ID:         AuthenticAMD
CPU family:        23
Model:             49
Model name:        AMD EPYC 7742 64-Core Processor
Stepping:          0
CPU MHz:           2245.780
BogoMIPS:          4491.56
Hypervisor vendor: KVM
Virtualization type:   full
L1d cache:         64K
L1i cache:         64K
L2 cache:          512K
L3 cache:          16384K
NUMA node0 CPU(s): 0-7
Flags:             fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 nt_good arat umip</pre>
I can execute YSQLSH to connect to my database in the following way. The <code>alter role</code> command changes the yugabyte superuser password. Do not let passwords retain their default value for anything that you care about:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>kubectl exec --namespace yb-demo -it yb-tserver-0 -- /home/yugabyte/bin/ysqlsh</b>
Defaulting container name to yb-tserver.
Use 'kubectl describe pod/yb-tserver-0 -n yb-demo' to see all of the containers in this pod.
Password for user yugabyte:
ysqlsh (11.2-YB-2.7.1.1-b0)
Type "help" for help.
yugabyte=# <b>select version();</b>
version
------------------------------------------------------------------------------------------------------------
 PostgreSQL 11.2-YB-2.7.1.1-b0 on x86_64-pc-linux-gnu, compiled by gcc (Homebrew gcc 5.5.0_4) 5.5.0, 64-bit
(1 row)
yugabyte=# <b>alter role yugabyte with password 'yb4oci4oke';</b>
ALTER ROLE
yugabyte=# \q
dev@cloudshell:~ (uk-london-1)$</pre>
Please mind the defined public endpoints. Like mentioned earlier, this is probably not a good idea for anything else than a demo cluster:
<pre class="lang:default decode:true"><code>dev@cloudshell:~ (uk-london-1)$ <b>kubectl --namespace yb-demo get services</b>
NAME                     TYPE               CLUSTER-IP         EXTERNAL-IP           PORT(S)                                                                          AGE

yb-master-ui             LoadBalancer   10.96.156.48   152.67.136.107        7000:30910/TCP                                                                   12m

yb-masters               ClusterIP          None               &lt;none&gt;                7000/TCP,7100/TCP                                                                12m

yb-tserver-service   LoadBalancer   10.96.94.161   140.238.120.201   6379:31153/TCP,9042:31975/TCP,5433:31466/TCP                                     12m

yb-tservers              ClusterIP          N</code></pre>
This means that I can connect to the YSQL port from any PostgreSQL client, like:
<pre class="lang:default decode:true">psql postgres://yugabyte:yb4oci4oke@140.238.120.201:5433/yugabyte</pre>
From <code>kubectl --namespace yb-demo get services</code> you see other ports. 5433 is the YSQL (PostgreSQL compatible) endpoint exposed by yb-tservers, through the load balancer. The whole cluster view is exposed by yb-masters on <a href="http://152.67.136.107:7000/" target="_blank" rel="noopener">http://152.67.136.107:7000</a> in my example.

Here is an example:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5tfu7vbt3qb89fqyay3g.png)

I’ve cut the screenshot so that you don’t see the code to reclaim a T-shirt but, if you install it, you can get it. ;) The read/write operations were limited by running the pgbench client from a free 1/8th OCPU but the 4 OCPU nodes can do more.
<h2 style="text-align: left">Kubectl</h2>
I find the Cloud Shell very handy for ad-hoc commands, but you may want to use oci-cli kubectl on your laptop (the below example is on Windows):

On Powershell, as administrator, install the OCI CLI:
<pre class="lang:default decode:true">md oke
cd oke
Set-ExecutionPolicy RemoteSigned
Invoke-WebRequest https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.ps1 -OutFile install.ps1
./install.ps1 -AcceptAllDefaults
cd ..
rd oke</pre>
On the Command Prompt, setup the connection to the cloud tenant:
<pre class="lang:default decode:true">oci setup config
oci setup repair-file-permissions --file C:\Users\franc\.oci\oci_api_key.pem
type oci_api_key_public.pem</pre>
The setup config asks for some OCID which you can get from the web console (upper-right icon):
<ul>
 	<li>Tenancy (<a href="https://cloud.oracle.com/tenancy" target="_blank" rel="noopener">https://cloud.oracle.com/tenancy</a>)</li>
 	<li>User Settings (<a href="https://cloud.oracle.com/identity/users" target="_blank" rel="noopener">https://cloud.oracle.com/identity/users</a>)</li>
</ul>
The public key generated must be added to the user, in User Settings “Add API Key”. I usually leave only 2 API keys as the maximum is 3 and I don't want to get blocked when I need to do something.
<h2 style="text-align: left">Cleanup</h2>
It is a lab. In order to reduce the cost when not used, you can scale down to zero nodes. And when you want to drop the cluster, you also need to delete the persistent volumes:
<pre class="lang:default decode:true">dev@cloudshell:~ (uk-london-1)$ <b>helm uninstall yb-demo -n yb-demo</b>
release "yb-demo" uninstalled
dev@cloudshell:~ (uk-london-1)$ <b>kubectl delete pvc --namespace yb-demo --all</b>
persistentvolumeclaim "datadir0-yb-master-0" deleted
persistentvolumeclaim "datadir0-yb-master-1" deleted
persistentvolumeclaim "datadir0-yb-master-2" deleted
persistentvolumeclaim "datadir0-yb-tserver-0" deleted
persistentvolumeclaim "datadir0-yb-tserver-1" deleted
persistentvolumeclaim "datadir0-yb-tserver-2" deleted
persistentvolumeclaim "datadir1-yb-master-0" deleted
persistentvolumeclaim "datadir1-yb-master-1" deleted
persistentvolumeclaim "datadir1-yb-master-2" deleted
persistentvolumeclaim "datadir1-yb-tserver-0" deleted
persistentvolumeclaim "datadir1-yb-tserver-1" deleted
persistentvolumeclaim "datadir1-yb-tserver-2" deleted
dev@cloudshell:~ (uk-london-1)$ <b>kubectl delete namespace yb-demo</b>
namespace "yb-demo" deleted</pre>
You can now Terminate the cluster. Note that if you didn’t do the previous cleanup, the block volumes and load balancer will stay. Here is my cost explorer when I was trying all that, creating and removing the cluster with terraform but without cleaning up the install:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v47aof8pz22j2re7i1h4.png)

Note that, beyond the price, leaving load balancers may prevent further creation as it is limited per-tenant.

This setup allows the creation of a Replication Factor 3 YugabyteDB universe, where rolling upgrades are possible because nodes can be restarted, one at a time. With Kubernetes, you can scale-out as needed and YugabyteDB manages the rebalancing of the shards automatically. Of course, for High Availability, you will have nodes in different Availability Domains (the AZ equivalent in OCI). The nice thing here is that the network traffic is free between Availability Domains in the Oracle Cloud. You can even have a node on your premises, and egress is free if you have FastConnect (the Direct Connect equivalent on OCI).

If you are in Europe in October, I’ll talk and demo this at HrOUG - the Croatian Oracle User Group: <a href="https://2021.hroug.hr/eng/Speakers/FranckPachot" target="_blank" rel="noopener">https://2021.hroug.hr/eng/Speakers/FranckPachot</a>.