The cost of OKE for YugabyteDB - 3 - Run a workload
Tags: yugabytedb, kubernetes, oracle, cloud
Published: 2022-04-11 14:54:04

The previous posts were about setting up an a [managed Kubernetes cluster](https://dev.to/franckpachot/the-cost-of-oke-for-yugabytedb-1-create-the-cluster-53m2) on the Oracle Cloud and [installing a YugabyteDB](https://dev.to/franckpachot/the-cost-of-oke-for-yugabytedb-2-install-from-helm-chart-b7l) database.

I want to generate some data and network traffic. I'll use my [ybdemo](https://github.com/FranckPachot/ybdemo/tree/main/docker/yb-lab/client) container to run some load. 

## Connect

The `connect` scenario is just connecting to the YugabyteDB cluster. I provide the `yb-tserver-service` LoadBalancer endpoint (the `hikari.properties` connects to `yb-tserver-0` and the `YBDEMO_DOMAIN` adds the kubernetes domain to it). This distributes the connections, but, in addition to that, I'm using the YugabyteDB cluster-aware JDBC driver and the connections are balanced to all pods. I could specify the specific zone in the connection string if I wanted to connect each application server locally.

```bash
dev@cloudshell:~ (uk-london-1)$ 

 kubectl -n yb-demo run ybdemo1 \
 --env="YBDEMO_DOMAIN=yb-tservers" --image=pachot/ybdemo \
 --env="YBDEMO_CASE=connect" --env="YBDEMO_THREADS=10"

pod/ybdemo1 created
```

The demo program shows where each thread is connected:
```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl -n yb-demo logs ybdemo1 | tail

Thread-20   1002 ms: 06-APR 09:46:10 pid:     2618    host:     10.244.0.132                                            
Thread-26   1003 ms: 06-APR 09:46:10 pid:     2652    host:       10.244.0.5                                            
Thread-17   1002 ms: 06-APR 09:46:10 pid:     2610    host:       10.244.0.5                                            
 Thread-8   1003 ms: 06-APR 09:46:10 pid:     2561    host:     10.244.0.132                                            
 Thread-2   1002 ms: 06-APR 09:46:10 pid:     2546    host:       10.244.0.5                                            
Thread-29   1002 ms: 06-APR 09:46:10 pid:     2659    host:     10.244.0.132                                            
 Thread-5   1002 ms: 06-APR 09:46:10 pid:     2647    host:     10.244.0.132                                            
Thread-23   1004 ms: 06-APR 09:46:10 pid:     2629    host:       10.244.1.4                                            
Thread-11   1002 ms: 06-APR 09:46:10 pid:     2595    host:     10.244.0.132                                            
Thread-14   1004 ms: 06-APR 09:46:10 pid:     2607    host:       10.244.1.4  
```
I have 3 pods for the moment, and I see the 3 IP addresses. It is easy to get their hostname to check where we are connected:
```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl get pods -n yb-demo 
 -o custom-columns="IP:.status.podIP,NAMESPACE:.metadata.namespace,NAME:.metadata.name"

IP             NAMESPACE   NAME
10.244.0.4     yb-demo     yb-master-0
10.244.1.3     yb-demo     yb-master-1
10.244.0.131   yb-demo     yb-master-2
10.244.0.132   yb-demo     yb-tserver-0
10.244.1.4     yb-demo     yb-tserver-1
10.244.0.5     yb-demo     yb-tserver-2
10.244.0.133   yb-demo     ybdemo1
```

With this configuration, one LoadBalancer redirects to any pod across my 3 Availability Domains. This load balancer is visible from the OCI console, but is managed by OKE:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e4oaj5kkt2sfnxms018o.png)

## Create

In order to do some reads and writes, I create a demo table with the `init scenario`:

```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl -n yb-demo run ybdemo0 --image=pachot/ybdemo \
 --restart=Never --env="YBDEMO_DOMAIN=yb-tservers" \
 --env="YBDEMO_CASE=init" --env="YBDEMO_THREADS=1"

pod/ybdemo0 created
```

The logs show the commands that have been run:
```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl -n yb-demo logs ybdemo0

drop table if exists demo;
ysqlsh:/dev/stdin:1: NOTICE:  table "demo" does not exist, skipping
DROP TABLE
create table if not exists demo(id bigint generated by default as identity, ts timestamptz default clock_timestamp(), message text, u bigint default 0, i timestamptz default clock_timestamp(), primary key(id hash));
CREATE TABLE
insert into demo(message) select format('Message #%s',generate_series(1,1000));
INSERT 0 1000
select format('All good :) with %s rows in "demo" table ',count(*)) from demo;
                   format                    
---------------------------------------------
 All good :) with 1000 rows in "demo" table 
(1 row)
```

If you want to check the tablets, the console displays them:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jppyphx8gub6zg0iuv52.png) 

## Inserts

I have a scenario that insert rows, and I'll run it from 3 pods with 300 threads each:
```bash
dev@cloudshell:~ (uk-london-1)$ 

 kubectl -n yb-demo run ybdemo2 --image=pachot/ybdemo \
 --env="YBDEMO_DOMAIN=yb-tservers" \
 --env="YBDEMO_CASE=insert" --env="YBDEMO_THREADS=300"

pod/ybdemo2 created

 kubectl -n yb-demo run ybdemo3 --image=pachot/ybdemo \
 --env="YBDEMO_DOMAIN=yb-tservers" \
 --env="YBDEMO_CASE=insert" --env="YBDEMO_THREADS=300"

pod/ybdemo3 created

 kubectl -n yb-demo run ybdemo4 --image=pachot/ybdemo \
 --env="YBDEMO_DOMAIN=yb-tservers" \
 --env="YBDEMO_CASE=insert" --env="YBDEMO_THREADS=300"

pod/ybdemo4 created
```
The logs show what is inserted, which traces the pod it is inserted from. This is the PostgreSQL backend process for the YugabyteDB session
```bash
dev@cloudshell:~ (uk-london-1)$ 

 kubectl -n yb-demo logs ybdemo2 | tail

Thread-40      9 ms: {"id":91458,"ts":"2022-04-06T09:56:04.397588+00:00","message":"inserted when connected to 10.244.0.5","u":0,"i":"2022-04-06T09:56:04.397596+00:00"}
Thread-45      6 ms: {"id":88799,"ts":"2022-04-06T09:56:04.40014+00:00","message":"inserted when connected to 10.244.0.132","u":0,"i":"2022-04-06T09:56:04.400182+00:00"}
Thread-18     10 ms: {"id":89492,"ts":"2022-04-06T09:56:04.396587+00:00","message":"inserted when connected to 10.244.0.132","u":0,"i":"2022-04-06T09:56:04.396594+00:00"}
Thread-10      7 ms: {"id":88991,"ts":"2022-04-06T09:56:04.39952+00:00","message":"inserted when connected to 10.244.0.132","u":0,"i":"2022-04-06T09:56:04.399529+00:00"}
Thread-33      8 ms: {"id":92627,"ts":"2022-04-06T09:56:04.398597+00:00","message":"inserted when connected to 10.244.0.5","u":0,"i":"2022-04-06T09:56:04.398604+00:00"}
Thread-27      8 ms: {"id":90371,"ts":"2022-04-06T09:56:04.400254+00:00","message":"inserted when connected to 10.244.0.132","u":0,"i":"2022-04-06T09:56:04.400263+00:00"}
Thread-48      8 ms: {"id":91357,"ts":"2022-04-06T09:56:04.40041+00:00","message":"inserted when connected to 10.244.0.132","u":0,"i":"2022-04-06T09:56:04.400418+00:00"}
 Thread-5      8 ms: {"id":92821,"ts":"2022-04-06T09:56:04.40054+00:00","message":"inserted when connected to 10.244.0.132","u":0,"i":"2022-04-06T09:56:04.400549+00:00"}
Thread-32      7 ms: {"id":93503,"ts":"2022-04-06T09:56:04.403381+00:00","message":"inserted when connected to 10.244.0.5","u":0,"i":"2022-04-06T09:56:04.403389+00:00"}
Thread-37      7 ms: {"id":92918,"ts":"2022-04-06T09:56:04.403144+00:00","message":"inserted when connected to 10.244.0.5","u":0,"i":"2022-04-06T09:56:04.403153+00:00"}
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l7qcx6oingzpjji9alog.PNG)

## Count

When I want to check the throughput I have a `count` scenario:
```bash
dev@cloudshell:~ (uk-london-1)$ 

 kubectl -n yb-demo run ybdemo5 --image=pachot/ybdemo --env="YBDEMO_DOMAIN=yb-tservers" \
 --env="YBDEMO_CASE=count" --env="YBDEMO_THREADS=1"

pod/ybdemo5 created
```
Note that it is better to have a range scan index on the timestamp, as the `count` query filters on the last minute:
```sql
yugabyte=# create index demots on demo(ts asc);
CREATE INDEX

/*+ IndexOnlyScan(demo demots) */ select format('Rows inserted in the last minute: %s',to_char(count(*),'999999999')) from demo where ts > clock_timestamp() - interval '1 minute';
```

The output shows the number of inserts per minute:
```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl -n yb-demo logs -f 

ybdemo5--------------------------------------------------
----- YBDemo -- Franck Pachot -- 2022-02-06 ------
----- https://github.com/FranckPachot/ybdemo -----
--------------------------------------------------
63 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
1672 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
--------------------------------------------------
sql executed in each new connection init:
--------------------------------------------------
prepare ybdemo(int) as select 
format('%8s pid: %8s %25s %30s %12s',to_char(now(),'DD-MON HH24:MI:SS') 
,pg_backend_pid(),'  host: '||lpad(host,16),cloud||'.'||region||'.'||zone,node_type) 
as yb_server, pg_sleep($1/1000) 
from (select replace(current_setting('listen_addresses'),'0.0.0.0',host(inet_server_addr())::text) as host) as server 
natural left join (select host,node_type,cloud,region,zone from yb_servers()) servers
--------------------------------------------------
input lines will start a thread to execute in loop
--------------------------------------------------

Starting a thread for    select format('Rows inserted in the last minute: %s',to_char(count(*),'999999999')) from demo where ts > clock_timestamp() - interval '1 minute'
 Thread-1   7894 ms: Rows inserted in the last minute:      60744
 Thread-1   6379 ms: Rows inserted in the last minute:      61462
 Thread-1   7890 ms: Rows inserted in the last minute:      60374
 Thread-1   7312 ms: Rows inserted in the last minute:      60264
 Thread-1   7394 ms: Rows inserted in the last minute:      60091
 Thread-1  10202 ms: Rows inserted in the last minute:      56947
```

## Scale out

The beauty of running a distributed database on Kubernetes is that scaling the StatefulSets is sufficient. YugabyteDB rebalances the connections, processing, and data.

The StatefulSets have 3 pods:

```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl get statefulsets -n yb-demo -o wide

NAME         READY   AGE   CONTAINERS              IMAGES
yb-master    3/3     66m   yb-master,yb-cleanup    yugabytedb/yugabyte:2.13.0.0-b42,yugabytedb/yugabyte:2.13.0.0-b42
yb-tserver   3/3     66m   yb-tserver,yb-cleanup   yugabytedb/yugabyte:2.13.0.0-b42,yugabytedb/yugabyte:2.13.0.0-b42
```

This is sufficient for `yb-master`, the control plane, but `yb-tserver` will benefit from scaling out. Let's bring them to 9 pods:

```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl scale -n yb-demo statefulset yb-tserver --replicas=9

statefulset.apps/yb-tserver scaled
```

Checking it (after a few minutes):
```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl get statefulsets -n yb-demo -o wide

NAME         READY   AGE   CONTAINERS              IMAGES
yb-master    3/3     67m   yb-master,yb-cleanup    yugabytedb/yugabyte:2.13.0.0-b42,yugabytedb/yugabyte:2.13.0.0-b42
yb-tserver   6/9     67m   yb-tserver,yb-cleanup   yugabytedb/yugabyte:2.13.0.0-b42,yugabytedb/yugabyte:2.13.0.0-b42
```

The YugabyteDB database detects automatically the new pods and re-balances the load:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l6rhczhrs87wtgqmnc7h.png)

The pods are distributed into the workers, thanks to the anti-affinity property. The workers are distributed to availability nodes by the node pool. The placement definition gives this information to YugabyteDB which re-balances the tablets leader and followers so that the quorum involves the 3 Availability Domains.
 
## Reads

My goal is to generate all kind of traffic. I have a `read` scenario that queries rows at random within a small work set:

```bash
dev@cloudshell:~ (uk-london-1)$ 

 kubectl -n yb-demo run ybdemo6 --image=pachot/ybdemo --env="YBDEMO_DOMAIN=yb-tservers" \
 --env="YBDEMO_CASE=read" --env="YBDEMO_THREADS=100"

pod/ybdemo6 created
```

## Rolling restart

The high availability of YugabyteDB is not only about infrastructure failures. We can restart the nodes in rolling fashion without interrupting the application. Here is an example:

```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl -n yb-demo rollout restart sts yb-tserver --wait

statefulset.apps/yb-tserver restarted
```

There is nothing more to do on the database side. YugabyteDB detects the nodes that fail, the quorum is still there and the application continues. The tablets that had their leader in the node that restarts have one of their followers becoming the leader. The threads that were connected to this node reconnects thanks to the connection pool and smart driver. When the pod restarts, the node gets the same volume and pulls the missed changes from the other tablet peers to synchronize the followers there. Then, some will be elected leaders to balance the load.

All nodes are busy, with connections, data, and taking reads and writes:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rakaiq4pkxt64dvbwvf1.png)

## Rolling update

The Helm chart installation defined 4GBbytes memory for the tserver pods:
```yaml
resource:
  master:
    requests:
      cpu: 2
      memory: 2Gi
    limits:
      cpu: 2
      memory: 2Gi
  tserver:
    requests:
      cpu: 2
      memory: 4Gi
    limits:
      cpu: 2
      memory: 4Gi
```
Given the worker VMs I have I want to increase it to 32GB.
This is easy with rolling update:
```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl -n yb-demo patch statefulset yb-tserver --type='json' -p='[ 
 {"op": "replace",
  "path": "/spec/template/spec/containers/0/resources/limits/memory",
  "value":"32Gi"},  
 {"op": "replace",
  "path": "/spec/template/spec/containers/0/resources/requests/memory",
  "value":"16Gi"} 
 ]'
```

This allows more memory for the container, which are available for connections (the PostgreSQL backend) but I also want the tserver to use more and this is fixed by the `yb-tserver` command line with `--memory_limit_hard_bytes=3649044480` from in the Helm template by `--memory_limit_hard_bytes={{ template "yugabyte.memory_hard_limit" $root.Values.resource.tserver }}`

I change it to 16GB with `kubectl edit sts yb-tserver -n yb-demo`:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m30cet4x1lp1e2n9bomj.png)

The containers restart with the new value:
```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl edit sts yb-tserver -n yb-demo

statefulset.apps/yb-tserver edited

dev@cloudshell:~ (uk-london-1)$

 kubectl get pods -n yb-demo -w

NAME           READY   STATUS              RESTARTS         AGE
yb-master-0    2/2     Running             0                27h
yb-master-1    2/2     Running             0                27h
yb-master-2    2/2     Running             0                27h
yb-tserver-0   2/2     Running             0                36m
yb-tserver-1   2/2     Running             0                36m
yb-tserver-2   2/2     Running             0                37m
yb-tserver-3   2/2     Running             0                38m
yb-tserver-4   2/2     Running             0                38m
yb-tserver-5   2/2     Running             0                39m
yb-tserver-6   2/2     Running             0                40m
yb-tserver-7   2/2     Running             0                40m
yb-tserver-8   0/2     ContainerCreating   0                2s
ybdemo2        1/1     Running             58 (4m14s ago)   26h
...
``` 

Once done, here is how I check the memory from the tserver memtrackers:
```bash
dev@cloudshell:~ (uk-london-1)$

 for i in yb-tserver-{0..8}
 do
  kubectl -n yb-demo exec -it $i -c yb-tserver -- \
   curl http://$i:9000/mem-trackers?raw |
  awk '{gsub(/<[/]?td>/," ")}/ (root) /{print n,$0}' n=$i
 done

yb-tserver-0      root  6.38G  6.42G  13.59G
yb-tserver-1      root  6.84G  6.86G  13.59G
yb-tserver-2      root  7.60G  7.63G  13.59G
yb-tserver-3      root  7.34G  7.37G  13.59G
yb-tserver-5      root  1.13G  1.13G  13.59G
yb-tserver-6      root  2.08G  2.24G  13.59G
yb-tserver-7      root  1.75G  2.17G  13.59G
yb-tserver-8      root  1.88G  2.24G  13.59G
```

Of course, I could have changed the resource spec while editing the StatefulSets and have only one restart.

## Terminate demo

If you want to terminate all those demo containers, to start it again, here is a quick loop:

```bash
dev@cloudshell:~ (uk-london-1)$

 kubectl get pods \
 -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name" \
 -A | grep ybdemo | while read namespace name 
 do 
  kubectl -n "$namespace" delete pod --force "$name" 
 done

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "ybdemo1" force deleted
```
 
## Uninstall

I'm letting this run to look at the cost after a few days. This will be in the next post. When you want to remove the cluster, you need to delete the resources. The compute instances will be automatically terminated, as they are part of the node pool, but the volumes and load balancers stay if you don't remove them. The instructions are in the install notes displayed at the end of the install.

I let this running and will add more workers. My goal, for the next post, is to look at the cost of this in the Oracle Cloud.