Repaving CockroachDB cluster node VMs the easy way
Tags: cockroachdb, tutorial
Published: 2023-01-30 21:27:26

In this brief post, I will demonstrate how you can repave/re-image your CockroachDB cluster VMs while still being online and without much data movement and replication.

The process is simple. For each node in the cluster:

1. Stop VM.
2. Detach data disks.
3. Upgrade/replace VM image with your new image (or use a new VM pre-configured with CockroachDB).
4. Attach disks to re-imaged/repaved VM (or the new VM).
5. Start VM and CockroachDB process.
6. Wait few minutes for cluster to adjust.

## Demo

I created a basic 3 nodes cluster on AWS with 1 attached EBS volume each. I started some sample load, and let it run for about 10 minutes.

![overview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c1gughe9fhpltky5q5px.png)

![sql-stmt-chart](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a3dutafkot2rswd8w2zt.png)

Here is the view of one of these nodes, see the mounted disk at `/dev/sdf`? That's an EBS volume.

![ec2-instance-description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ezl0h1rh9kddqwtjz1e.png)


I then reached out to the AWS Console and stopped a node, `n3`. CockroachDB complains about a suspect/dead node, and shows that some ranges are under-replicated.

![overview-underreplication](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1hh7wdbo27kjbpwf52xo.png)

Once the VM was stopped, I detached the EBS volume (see green banner)... 

![vol-detached](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pq4ak74rbwy0o8u407y6.png)

...and attached it to a new and up and running VM.

![attach-vol](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ich056deg0tany5nl0xr.png)

I then ssh'ed into the new VM, and started the CockroachDB process. If you are repaving into a **new** VM, make sure the [TLS certificates for inter-node communication](https://www.cockroachlabs.com/docs/v22.2/security-reference/transport-layer-security.html#tls-between-cockroachdb-nodes) are correctly installed.
Immediately, the node connects to the cluster and complains about under-replicated ranges are gone.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ofrs04tan1cbwhoogxz.png)

That's it! Confirm under-replicated ranges are gone, then wash, rinse and repeat for every other node!

While, as you can see from the charts, the whole process took about 5 minutes, this can be greatly simplified and improved upon by using a dev-ops automation tool such as Ansible. 

### Cleaning up the old nodes

Once repaved, we can remove the notion of the old, repaved node `n3` from the cluster using the [decommission](https://www.cockroachlabs.com/docs/stable/cockroach-node.html#node-decommission) command.

```bash
root@ip-10-10-11-95:/home/ubuntu# cockroach node status --certs-dir /var/lib/cockroach/certs/ 
  id |       address       |     sql_address     |  build  |              started_at              |              updated_at              |        locality         | is_available | is_live
-----+---------------------+---------------------+---------+--------------------------------------+--------------------------------------+-------------------------+--------------+----------
   1 | 18.224.59.170:26257 | 18.224.59.170:26257 | v22.2.3 | 2023-01-30 20:03:59.66018 +0000 UTC  | 2023-01-30 20:26:07.183524 +0000 UTC | region=us-east-2,zone=a | true         | true
   2 | 18.119.133.34:26257 | 18.119.133.34:26257 | v22.2.3 | 2023-01-30 20:04:00.13035 +0000 UTC  | 2023-01-30 20:26:07.642953 +0000 UTC | region=us-east-2,zone=a | true         | true
   3 | 18.117.158.82:26257 | 18.117.158.82:26257 | v22.2.3 | 2023-01-30 20:04:00.944523 +0000 UTC | 2023-01-30 20:13:52.332716 +0000 UTC | region=us-east-2,zone=a | false        | false
   4 | 3.141.0.11:26257    | 3.141.0.11:26257    | v22.2.3 | 2023-01-30 20:21:37.594691 +0000 UTC | 2023-01-30 20:26:07.609098 +0000 UTC | region=us-east-2,zone=a | true         | true
(4 rows)
root@ip-10-10-11-95:/home/ubuntu# 
root@ip-10-10-11-95:/home/ubuntu# 
root@ip-10-10-11-95:/home/ubuntu# cockroach node decommission 3 --certs-dir /var/lib/cockroach/certs/ 

  id | is_live | replicas | is_decommissioning |   membership    | is_draining
-----+---------+----------+--------------------+-----------------+--------------
   3 |  false  |        0 |        true        | decommissioning |    true
(1 row)

No more data reported on target nodes. Please verify cluster health before removing the nodes.
```

![decommissioned-nodes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y9o6gb9po0wstgjrd1us.png)


The cluster is now updated and as good as new.

## References

- [CockroachDB](https://www.cockroachlabs.com/product/)
- [Node Shutdown](https://www.cockroachlabs.com/docs/v22.2/node-shutdown)