YugabyteDB Helm Chart services: LoadBalancer and headless ClusterIP ☸️🚀
Tags: yugabytedb, kubernetes, database, postgres
Published: 2023-01-17 22:00:52

Installing YugabyteDB with the Helm Chart creates 4 services. Here is an example where I deployed in AWS as in this [previous blog post](https://dev.to/aws-heroes/yugabytedb-on-amazon-eks-3206):
```sh
$ kubectl get services -n yb-demo-eu-west-1a

NAME                 TYPE           CLUSTER-IP       PORT(S)
yb-master-ui         LoadBalancer   10.100.227.70    7000:31261/TCP
yb-masters           ClusterIP      None             7000/TCP,7100/TCP
yb-tserver-service   LoadBalancer   10.100.31.106   6379:31669/TCP,9042:31874/TCP,5433:31376/TCP
yb-tservers          ClusterIP      None             9000/TCP,12000/TCP,11000/TCP,13000/TCP,9100/TCP,6379/TCP,9042/TCP,5433/TCP
```

There are two services (LoadBalancer and ClusterIP) for the two StatefulSets (`yb-master`, the contol plane, and `yb-tservers`, the data plane)

## ClusterIP headless service

The **ClusterIP** ones are created with no IP which means that they are just DNS entries and not a proxy. This is a **headless service** that can be used to distribute the connections without an intermediate component, using the Kubernetes DNS.

From the cluster (`cluster.local`) in a namespace (`yb-demo-eu-west-1c`), I can connect to any pod in the `yb-tserver` StatefulSets with the hostname `yb-tservers.yb-demo-eu-west-1a.svc.cluster.local`

## Round robin to one StatefulSets pods

Here is an example, running `psql` from any pod, and connecting multiple times to this address:

```sql
$ kubectl run -it --rm --restart=Never --image postgres psql -- \
  psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local      \
  -p 5433 -U yugabyte -c 'select inet_server_addr()'

 inet_server_addr
------------------
 192.168.11.13
(1 row)

$ kubectl run -it --rm --restart=Never --image postgres psql -- \
  psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local      \
  -p 5433 -U yugabyte -c 'select inet_server_addr()'

 inet_server_addr
------------------
 192.168.20.199
(1 row)
```

The connections went to different nodes. Those are different PostgreSQL backends running in different pods, but they expose the same logical database because YugabyteDB is a **Distributed SQL** database.

The two IP addresses above are the two pods in the `yb-tserver` StatefulSets:

```sh
$ kubectl get pods -n yb-demo-eu-west-1a -o wide

NAME           READY   STATUS    RESTARTS   AGE    IP               NODE                                          NOMINATED NODE   READINESS GATES
yb-master-0    2/2     Running   0          174m   192.168.2.75     ip-192-168-4-117.eu-west-1.compute.internal   <none>           <none>
yb-tserver-0   2/2     Running   0          174m   192.168.20.199   ip-192-168-4-117.eu-west-1.compute.internal   <none>           <none>
yb-tserver-1   2/2     Running   0          174m   192.168.11.13    ip-192-168-4-117.eu-west-1.compute.internal   <none>           <none>
```

Note that I deployed the StatefulSets in three Availability Zones. The ClusterIP service connects to one AZ and this is probably what you want to do from the application servers, so that they connect in the same AZ. This ensures the minimum latency and maximum availability.

## Cluster aware Smart Drivers

However, if you use the YugabyteDB [Smart Drivers](https://docs.yugabyte.com/preview/drivers-orms/smart-drivers/) they will discover all nodes in all AZ:

```sql
$ kubectl run -it --rm --restart=Never --image postgres psql -- \
>   psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local      \
>   -p 5433 -U yugabyte -c 'select host,zone from yb_servers()'
                             host                              |    zone
---------------------------------------------------------------+------------
 yb-tserver-0.yb-tservers.yb-demo-eu-west-1c.svc.cluster.local | eu-west-1c
 yb-tserver-1.yb-tservers.yb-demo-eu-west-1a.svc.cluster.local | eu-west-1a
 yb-tserver-0.yb-tservers.yb-demo-eu-west-1b.svc.cluster.local | eu-west-1b
 yb-tserver-1.yb-tservers.yb-demo-eu-west-1b.svc.cluster.local | eu-west-1b
 yb-tserver-1.yb-tservers.yb-demo-eu-west-1c.svc.cluster.local | eu-west-1c
 yb-tserver-0.yb-tservers.yb-demo-eu-west-1a.svc.cluster.local | eu-west-1a
(6 rows)
```
This mean that if you want to constrain the connections to one AZ when using the Smart Drivers, you need to use the tolopogy key, like `aws.eu-west-1.eu-west-1a` for example.

## Endpoints

The ClusterIP headless service `yb-tservers` connects to the `yb-tserver` pods, 192.168.11.13 and 192.168.20.199 in my case, for all ports exposed by the Table Servers, 5433 being the YSQL one, which is the **PostgreSQL-compatible** API.

```sh
$ kubectl describe service yb-tservers -n yb-demo-eu-west-1a

Name:              yb-tservers
Namespace:         yb-demo-eu-west-1a
Labels:            app=yb-tserver
                   app.kubernetes.io/managed-by=Helm
                   chart=yugabyte
                   component=yugabytedb
                   heritage=Helm
                   release=yb-demo
                   service-type=headless
Annotations:       meta.helm.sh/release-name: yb-demo
                   meta.helm.sh/release-namespace: yb-demo-eu-west-1a
Selector:          app=yb-tserver
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              http-ui  9000/TCP
Endpoints:         192.168.11.13:9000,192.168.20.199:9000
Port:              http-ycql-met  12000/TCP
TargetPort:        12000/TCP
Endpoints:         192.168.11.13:12000,192.168.20.199:12000
Port:              http-yedis-met  11000/TCP
TargetPort:        11000/TCP
Endpoints:         192.168.11.13:11000,192.168.20.199:11000
Port:              http-ysql-met  13000/TCP
TargetPort:        13000/TCP
Endpoints:         192.168.11.13:13000,192.168.20.199:13000
Port:              tcp-rpc-port  9100/TCP
TargetPort:        9100/TCP
Endpoints:         192.168.11.13:9100,192.168.20.199:9100
Port:              tcp-yedis-port  6379/TCP
TargetPort:        6379/TCP
Endpoints:         192.168.11.13:6379,192.168.20.199:6379
Port:              tcp-yql-port  9042/TCP
TargetPort:        9042/TCP
Endpoints:         192.168.11.13:9042,192.168.20.199:9042
Port:              tcp-ysql-port  5433/TCP
TargetPort:        5433/TCP
Endpoints:         192.168.11.13:5433,192.168.20.199:5433
Session Affinity:  None
Events:            <none>
```

## LoadBalancer

In addition to the ClusterIP service, the Helm Chart creates a LoadBalancer. I can see it in the AWS console:

![console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1wcq6hq5r8i5wo5mb3v4.png)

![instances](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cwy5vnpssoidlks39ps8.png)

Here is the description from Kubernetes:

```sh
$ kubectl describe service yb-tserver-service -n yb-demo-eu-west-1a

Name:                     yb-tserver-service
Namespace:                yb-demo-eu-west-1a
Labels:                   app=yb-tserver
                          app.kubernetes.io/managed-by=Helm
                          chart=yugabyte
                          component=yugabytedb
                          heritage=Helm
                          release=yb-demo
                          service-type=endpoint
Annotations:              meta.helm.sh/release-name: yb-demo
                          meta.helm.sh/release-namespace: yb-demo-eu-west-1a
Selector:                 app=yb-tserver
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.31.106
IPs:                      10.100.31.106
LoadBalancer Ingress:     a95638155644e470abd19e552bc8ab01-1510239497.eu-west-1.elb.amazonaws.com
Port:                     tcp-yedis-port  6379/TCP
TargetPort:               6379/TCP
NodePort:                 tcp-yedis-port  31498/TCP
Endpoints:                192.168.11.13:6379,192.168.20.199:6379
Port:                     tcp-yql-port  9042/TCP
TargetPort:               9042/TCP
NodePort:                 tcp-yql-port  30927/TCP
Endpoints:                192.168.11.13:9042,192.168.20.199:9042
Port:                     tcp-ysql-port  5433/TCP
TargetPort:               5433/TCP
NodePort:                 tcp-ysql-port  31565/TCP
Endpoints:                192.168.11.13:5433,192.168.20.199:5433
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
```

The **ingress** host is the one I can use to connect from outside of the Kubernetes cluster. In this Multi-AZ configuration I have one per AZ. Let's check the round-robin with 100 sucessive connections:

```sh
$ for i in {1..100} ; do
   psql -h a95638155644e470abd19e552bc8ab01-1510239497.eu-west-1.elb.amazonaws.com -p 5433 \
   -t -A -c '
     select inet_server_addr()
    ' ; done | sort | uniq -c

     52 192.168.11.13
     48 192.168.20.199

```
48% on yb-tserver-0 and 52% on yb-tserver-1 

To create StatefulSets for **Multi-AZ** (where each has a `yb-master` and knows about all `yb-master` in the cluster) I used an override with `isMultiAz: True`. If you don't want the LoadBalancer, you can disable it with ` enableLoadbalancer: false`

Those values are visible with:
```sh
helm show all yugabytedb/yugabyte
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wzkublgnaqyk5mnlylz0.png)
and if you don't ind the documentation and comments sufficient, the template is: [yugabyte/charts service.yaml](https://github.com/yugabyte/charts/blob/master/stable/yugabyte/templates/service.yaml#L101)


Note that Helm Charts are the maintained and recommended way to Install YugabyteDB on Kubernetes. There's no need for **operators** as the database itself is **self-healing**, autonomously repairing itself autonomously without the need for additional components. You scale by scaling the StatefulSets and the connections, SQL processing and data are rebalanced automatically. When a pod is down, the remaining ones do the necessary Raft Leader election to continue to serve consistent reads and writes.