CockroachDB: Multi-Region OpenShift using Azure Virtual WAN
Tags: cockroachdb, azure, openshift
Published: 2023-08-10 16:25:44

### _...art of the practical_ ###

Networking between managed OpenShift clusters has become practical through Azure's **_Virtual WAN_**. When deploying **CockroachDB** on top, you become the owner of a production-ready, globally available distributed database. Let's create the key components that facilitate secure multi-region networking and architect this database solution.

### what is this? ###

This blog features a single logical CockroachDB database that’s deployed on **_ARO_** (Azure Red Hat OpenShift) clusters which are provisioned across multiple regions.

This article is my second instalment into OpenShift networking at global scale. Here we leverage **Microsoft Azure**, and this complements my original post [CockroachDB: Multi-Region ROSA using Cloud WAN](https://dev.to/world2mark/cockroachdb-multi-region-rosa-using-cloud-wan-2h39) that focuses on **AWS** equivalent infrastructure.

### the technical catalog ###

- **Virtual Networks** are the fundamental building blocks for private networks in Azure. This foundation provides the ability for cloud services to interconnect and securely communicate with each other.
- **Subnets** are the specific IP addresses range constructs where resources are grouped and interconnected within a virtual network.
- **ARO** is the highly available, fully managed service for deploying Red Hat OpenShift instances that are monitored and operated jointly by Microsoft and Red Hat. OpenShift is Kubernetes running on Red Hat Enterprise Linux (RHEL) with built-in management UI services for convenience.
- **Virtual WAN** is a networking service that brings many networking, security, and globally scaled routing functionalities together, managed by a single operational interface. This is the key offering that lets us manage and connect local and regional **_virtual networks_** together.
- **Virtual HUB** is the service that lets you connect virtual networks together across sites and regions. The Virtual WAN architecture is a hub and spoke model so at least 1 hub is always required to establish a network-service mesh for your resources across the Microsoft network backbone.
- **CockroachDB** (aka _**CRDB**_) is the distributed database that is deployed on the worker nodes across the OpenShift clusters.

### the typical ARO cluster ###

It is important to have a background into the relevant components of a managed Red Hat OpenShift cluster to understand how they can be connected together. When creating an ARO cluster, the resources and services are pre-configured and automatically deployed based on the sizing criteria that is specified during the creation process. Virtual Network **Subnets** are the required critical networking aspect of ARO clusters that compartmentalize the master and worker node pools. These are used by the Cloud WAN service and Hubs to establish the network routing rules.

![ARO Cluster in Virtual Network](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/asn1i22d6ib4ykv53jka.png)
<figcaption>Architecture: This managed OpenShift cluster hosts containerized apps including CockroachDB nodes. Overview of <a href="https://azure.microsoft.com/en-ca/products/openshift">Microsoft Azure ARO</a></figcaption>

----

### Virtual WAN: facilitator of OpenShift cluster networking ###

This service defines the networking infrastructure and connectivity rules for the complete solution. The **Virtual WAN** defines the core network-framework that manages hubs, VPN sites, EspressRoute circuits, and virtual networks. My searches online revealed many great examples of single and multi-hub networks that handle groups of networks, including how to establish links to on-prem environments or other cloud resources (eg: multi cloud solutions).

In our solution, we will use a single **HUB** to define our CockroachDB network, and a distinct **Virtual Network** connections that’s associated with every managed OpenShift cluster in our solution.

The most important caveat is to define unique subnet ranges for each OpenShift environment, as this is how we’ll differentiate and access the actual network IPs across each node in each cluster.

![Multiple ARO Clusters interconnected via Virtual WAN](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f8x43egsnuzd50wt3ndq.png)
<figcaption>Architecture: Virtual WAN allows interconnectivity between ARO clusters across regions, resulting in a network mesh that’s ideal for a distributed CockroachDB platform. Note the unique CIDR ranges for each virtual network.</figcaption>

----

### the network service mesh ###

- In the following diagram, all master nodes are hidden for simplicity.  They are typically hosted on the 172.x.0.x/16 subnet, while all workers are hosted on the 172.x.1.x/16 subnet.
- Worker nodes within each cluster are inherently connected; links have been omitted for simplicity in this chart.
- Every node has network visibility to every other node across clusters and regions. This example only illustrates Node 2 links for simplicity.

![Network service mesh](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1h29ygstu13je1wtpfyc.png)
<figcaption>Architecture: The CockroachDB network mesh requires that every worker node hosting a CockroachDB pod has routing/visibility to every other worker node (by extension every other CockroachDB pod) in all the OpenShift clusters across regions.</figcaption>

----

### let’s create the Azure resources ###

The following steps will establish the necessary cluster infrastructure based on the included screenshots, but your settings may vary, such as IP CIDR ranges or regional settings.

#### Step 1: Create your Virtual Networks ####

For each OpenShift cluster, you need a corresponding **virtual network** with 2 subnets.  One subnet is for OpenShift workers, the other is for OpenShift masters. This is required for associating an ARO cluster with a valid virtual network during the creation process.
Below is my new virtual network, named **zlamal-aro-canada-01** with the 2 subnets.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c0tbl50x9f2uxzfxy5vv.png)
<figcaption>I use default as my masters subnet, and create the second workers subnet with a uniquely routable CIDR range.</figcaption>

![]()<figcaption></figcaption>

#### Step 2: Create your OpenShift Clusters ####

We will focus on the networking aspects of the Azure ARO _Creation Wizard_.
After specifying the cluster name, region and node sizing, the **Virtual Networks** list will be filtered to the same region of the cluster region.  In my example I chose Canada Central for my Cluster & networking. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/877s7utumgfmb4wd9kfr.png)<figcaption>Master subnet is my default virtual network subnet, while the Worker subnet is tied to my worker-virtual-network subnet.</figcaption>

I’ve chosen to make the endpoints **visible** for convenience and to avoid the challenges of establishing a VPN framework for this exercise.
> Notice: This screenshot indicates **172.18.x.x** virtual network subnets while the prior screenshot shows **172.23.x.x**. This is because during this blog write-up I’ve created yet another virtual network for an upcoming OpenShift cluster, but normally these CIDR ranges would align with each other.

#### Step 3: Create a Virtual WAN instance ####

In this diagram, the first step is to specify the management region of the Cloud WAN instance.  Actual networking at this phase is agnostic of locality.  Specify your name and ensure the **Type** is **Standard**.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r0gc68dqh4zfuwywyrov.png)

![]()<figcaption></figcaption>

#### Step 4: Create the Hub instance ####

This hub resides in the Virtual WAN, and will connect all the OpenShift clusters together in the subsequent step. The hub is also assigned a region, but ultimately accepts connections from any Azure network-resource provisioned in any region.

In this example it’s named _**canada-east-hub**_, along with a special-defined address space used internally by Microsoft to manage/operate the hub instance.  I’ve chosen a non-overlapping CIDR of **172.100.0.0/16** and this will not interfere with my clusters or any future additions to this network. To save on costs I kept the hub capacity at the minimal possible throughput and connection capability.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j2t0qrvspo1vw2iqa2p3.png)<figcaption>”Basics” tab is the only one that’s relevant.  All others can be kept as default.</figcaption>

![]()<figcaption></figcaption>

#### Step 5: Add the connections to the Hub ####

Every OpenShift cluster resides on a unique Virtual Network. This step establishes the connectivity on all the virtual networks. The hub manages and operates this pool of connections.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/34lqpwmt4b5k7jvegcz8.png)<figcaption>To add a connection, select your Virtual WAN, under _Virtual network connections_ click on **Add connection**. From here you can label this interface, apply the Hub, apply default settings, and complete the peering.</figcaption>

![]()<figcaption></figcaption>

> As you scale-out, create, or delete OpenShift clusters, here is where the interfaces are established or removed.  This means creation of a Virtual WAN along with the Hub is a one-time process.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gm3f8950wxgwfez2acye.png)<figcaption>Upon completion of virtual network associations with the hub, the network topology will be updated with your complete network.</figcaption>

![]()<figcaption></figcaption>

#### Step 6: Establish CockroachDB nodes ####

Every OpenShift is interconnected at this point. This means that if you open a terminal window to any running pod, you can **curl** to any worker on any of the clusters.

- From this point you can create a unique OpenShift deployment instance for each worker-node affinity to ensure a 1:1 relationship between workers and CRDB nodes.

- Catalog the IP addresses of the worker nodes in your _**primary**_ OpenShift cluster that will serve as the starting point for CockroachDB.  Once this primary cluster of 3+ nodes is initialized, all the other nodes can use these 3+ IP addresses, along with corresponding service **NodePorts** to connect to every other node on the cluster.

- Load balancers can be created per-OpenShift-cluster, providing additional redundancy if a cluster becomes unavailable.  This also clearly defines multiple access-points for clients to use and prioritize based on latency and geographic location.

This YAML fragment illustrates the locality and network parameters for a single node in the cluster. The _**--join**_ statement is typically static for all nodes, while the sql-addr, listen-addr, and advertise-addr will be unique for each node. 

```
          ...
          ...
apiVersion: apps/v1
kind: Deployment
metadata:
  name: crdb-deployment-zlamal-172-23-1-4
  labels:
    crdb-cluster: crdb-zlamal
spec:
  selector:
    matchLabels:
      crdb-pod-name: crdb-pod-zlamal-172-23-1-4
  replicas: 1
  template:
    metadata:
      labels:
        crdb-pod-name: crdb-pod-zlamal-172-23-1-4
        crdb-cluster: crdb-zlamal
    ...
    spec:
      ...
      containers:
          command:
            - /bin/bash
            - '-ecx'
            - >-
              exec /cockroach/cockroach
              start
              --cache=.25
              --logtostderr
              --certs-dir=/cockroach/cockroach-certs
              --locality=country=CA,region=CA-East,zone=Zone-1
              --sql-addr=:26257
              --listen-addr=:26357
              --advertise-addr=172.23.1.4:31951
              --join 172.23.1.6:31950,172.23.1.4:31951,172.23.1.5:31952
              ...
              ...
```

Each CRDB node will require its own _**NodePort**_ service to establish the internal database network mesh, while a common service can internally load-balance connections for co-located applications.

> This example service aligns with the above fragment to define unique NodePort port numbers that must not overlap with other services in the current OpenShift cluster.

```
kind: Service
apiVersion: v1
metadata:
  name: crdb-service-zlamal-172-23-1-4
spec:
  ports:
    - name: sql-access
      protocol: TCP
      port: 26257
      targetPort: 26257
      nodePort: 31851
    - name: node-comms
      protocol: TCP
      port: 26357
      targetPort: 26357
      nodePort: 31951
    - name: console-ui
      protocol: TCP
      port: 8080
      targetPort: 8080
  type: NodePort
  selector:
    crdb-pod-name: crdb-pod-zlamal-172-23-1-4
```

> A generic service would have the _**selector**_ not to a pod, but rather the common label of the deployments. In this case set the selector to **crdb-zlamal** for the load balancers to properly interface with all the active pods.

For example a load balancer service definition handing the local cluster will have a selector as follows:

```
apiVersion: v1
kind: Service
metadata:
  name: crdb-zlamal-load-balancer
  labels:
    crdb-cluster: crdb-zlamal
spec:
  ports:
    - name: console-ui
      port: 8080
      protocol: TCP
      targetPort: 8080
    - name: sql-access
      port: 26257
      protocol: TCP
      targetPort: 26257
  type: LoadBalancer
  selector:
    crdb-cluster: crdb-zlamal
```

----

### conclusion ###

These Azure network services abstract away the complexities of infrastructure, security, and resource-management, offering a view of the entire architecture from a single pane of glass.  I can't cover every detail into these technologies, but Azure also provides service & network monitoring, security controls, and visual representations of the connected ecosystem.

For a starter-pack of Kubernetes templates to help with deploying CockroachDB, go to the [Cockroach/cloud/kubernetes](https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes) GitHub repo.

----

### references ###

[Cockroach  Labs](https://www.cockroachlabs.com)
[CockroachDB: Multi-Region ROSA (AWS) using Cloud WAN](https://dev.to/world2mark/cockroachdb-multi-region-rosa-using-cloud-wan-2h39)
[What is Azure Virtual WAN?](https://learn.microsoft.com/en-us/azure/virtual-wan/virtual-wan-about)
[Azure ARO](https://azure.microsoft.com/en-ca/products/openshift)