Aerospike k8s Volume Cleanup
Tags: aerospike, developers, devops, kubernetes
Published: 2022-10-13 21:33:16

​
<div className="text--center">

![](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/disk-image_1663780313517.png)
​
</div>

When using the Aerospike Kubernetes Operator, the complexity of configuring a  high performance distributed database is abstracted away, making the task of instantiating an Aerospike database incredibly easy. However, even though Kubernetes leads us towards expecting equivalent results regardless of platform, we need to be mindful of the peculiarities of individual frameworks, particularly if we are repeatedly iterating processes. 
​
This article focuses on AWS EKS provisioned storage which is dynamically created when using the Aerospike Kubernetes Operator. Ensuring that storage has been fully deleted and other redundant resources removed is a necessary housekeeping step if you are to avoid unwelcome AWS charges.
​
<!--truncate-->
​
In this article we review
​
+ How to create a simple AWS EKS cluster
+ Install the Aerospike Kubernetes Operator
+ How AWS Provisioned EBS Storage volumes are created
+ The results of repeated Aerospike database cluster creation - particularly the resulting storage utilization
+ Database cluster termination, with a focus on how the storage assets are managed
+ What to watch out for

---

## Create an EKS Cluster
​
In this section we create an EKS cluster, on which we will install Aerospike.
​
### Setup
​
Create a file called *my-cluster.yaml* with the following contents. This specifies the configuration for the Kubernetes cluster itself. The *ssh* configuration is optional.

```
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
​
metadata:
  name: my1-eks-cluster
  region: us-east-1
​
nodeGroups:
  - name: ng-1
    labels: { role: database }
    instanceType: t3.medium
    desiredCapacity: 3
    volumeSize: 13
    ssh:
      allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key
```
We use the AWS eksctl tool to create our Kubernetes cluster. For more information on using eksctl see the official documentation.
​
[![eksctl - The official CLI for Amazon EKS](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/eksctl_1663848300911.png)](https://eksctl.io/)
​
Using *eksctl* ...
```
eksctl create cluster -f my-cluster.yaml
```
The result on completion will be a 3 node Kubernetes cluster. Confirm successful creation as follows :
```
kubectl get nodes -L role
NAME                             STATUS   ROLES    AGE   VERSION                ROLE
ip-192-168-13-47.ec2.internal    Ready    <none>   10m   v1.22.12-eks-ba74326   database
ip-192-168-23-112.ec2.internal   Ready    <none>   10m   v1.22.12-eks-ba74326   database
ip-192-168-40-59.ec2.internal    Ready    <none>   10m   v1.22.12-eks-ba74326   database
```

We can check what volumes are created. In *my-cluster.yaml* volume size is 13Gb, allowing these to be easily located when using the AWS EC2 console.

![](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/eks-ebs-volumes-k8s-cluster_1663767228439.png)

---

## Install Aerospike
### Aerospike Kubernetes Operator
​
In this section we install the AKO ( Aerospike Kubernetes Operator ) and set up our database cluster. 
​
Please refer to the following documentation for details as to how to do this.
​
[![AKO Install Docs](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/ako-install-docs_1663846737344.png)](https://docs.aerospike.com/cloud/kubernetes/operator/install-operator-operatorhub)

AKO installation can be confirmed by checking the pods in the *operators* namespace.

```
kubectl get pods -n operators
NAME                                                     READY   STATUS    RESTARTS   AGE
aerospike-operator-controller-manager-7946df5dd9-lmqvt   2/2     Running   0          2m44s
aerospike-operator-controller-manager-7946df5dd9-tg4sh   2/2     Running   0          2m44s
```

### Create the Aerospike Database Cluster
​
In this section we create an Aerospike cluster. For full detail please refer to the documentation below.
​
[![Aerospike Cluster](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/create-cluster-ako-2_1663846950543.png)](https://docs.aerospike.com/cloud/kubernetes/operator/create-cluster-kubectl#prepare-the-aerospike-cluster-configuration)
​
The basic steps are as follows.
​
#### Get the code
Clone the Aerospike Kubernetes Git repo and be sure to copy your feature/licence key file, if you have one, to the following directory - *config/samples/secrets*.
```
git clone https://github.com/aerospike/aerospike-kubernetes-operator.git
cd aerospike-kubernetes-operator/
```

#### Initialise Storage
```
kubectl apply -f config/samples/storage/eks_ssd_storage_class.yaml
kubectl apply -f config/samples/storage/local_storage_class.yaml
```

#### Add Secrets
​
Secrets, for those not familiar with the terminology, allow data to be introduced into the Kubernetes environment, while ensuring that it canot be read. Examples might include private PKI keys or passwords.
​
```
kubectl  -n aerospike create secret generic aerospike-secret --from-file=config/samples/secrets
kubectl  -n aerospike create secret generic auth-secret --from-literal=password='admin123'
```

#### Create Aerospike Cluster
​
We set up Aerospike using the [ssd storage](https://docs.aerospike.com/server/operations/configure/namespace/storage#recipe-for-an-ssd-storage-engine) recipe.
​
```
kubectl create -f config/samples/ssd_storage_cluster_cr.yaml 
```

#### Verify Aerospike Database Running
The `get pods` command should show you two active Aerospike pods.
```
kubectl get pods -n aerospike
NAME              READY   STATUS    RESTARTS   AGE
aerocluster-0-0   1/1     Running   0          36s
aerocluster-0-1   1/1     Running   0          36s
```

---

## Persistent Volumes and Claims (PV,PVC)
If we run the following command, we can see what persistent volume storage has been claimed.

```
kubectl get pv,pvc -n aerospike -o json | jq .'items[].spec.capacity.storage' | egrep -v -e null
"1Gi"
"5Gi"
"5Gi"
"1Gi"
```
​
Note we see the same persistent volumes when using the AWS Console as expected -  included for completeness.

![](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/eks-ebs-volumes-k8s-cluster-plus-pvc_1663773743635.png)

---

## Cleaning up
​
**WARNING:** If this step is bypassed and the EKS cluster is deleted, all persistent volumes created will remain.

### Delete Aerospke Database Cluster
***This*** is the important step and should always be run when using cloud storage classes.
​
```
kubectl delete -f ssd_storage_cluster_cr.yaml 
aerospikecluster.asdb.aerospike.com "aerocluster" deleted
```

### Check Persistent Volume Claims
If we now check our persistent volumes we see there are no persistent volumes remaining.

```
kubectl get pv,pvc -n aerospike -o json | jq .'items[].spec.capacity.storage' | egrep -v -e null
(nothing returned...)
```
​
Similarly, if visiting the AWS Console, we can see only the volumes for the k8s nodes we initially created using the *eksctl* command.
​

![](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/eks-ebs-volumes-k8s-cluster-minus-pvc_1663774458841.png)
​
### Delete EKS Cluster
We can now go ahead and delete the EKS cluster

```
eksctl delete cluster -f my-cluster.yaml
```

### Verify Resources Deleted
Final check of our EBS volumes and we should find everything has been removed.
​

![](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/ebs-eks-volumes-left_1663775144130.png)

---

## How *not* to clean up
​
Assume we have a new EKS cluster, having installed the Aerospike Operator and a new Aerospike database. We then see the resources below.

![](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/eks-ebs-volumes-k8s-cluster-plus-pvc_1663773743635.png)

​
Suppose we *only* we excecute the `eksctl delete cluster` command.

```
eksctl delete cluster -f my-cluster.yaml
```

![](https://developer-hub.s3.us-west-1.amazonaws.com/naresh-maharaj/ebs-eks-volumes-not-removed_1663778929910.png)

In that case, the persistent volumes associated with Aerospike will ***NOT*** be deleted. In this example the volumes are small, but at $0.10/GB/month these costs can accumulate (100TB = $10k / month!). Numerous *test* iterations can result in capacity accumulating, generating large unwelcome AWS bills for.
​
Note that we can go ahead and manually delete these volumes via the AWS Console - but only because we know they are there.

## Conclusion

Automation tools are fantastic and this level of deployment agility is an astonishing convenience. It is always wise however to check on final results and not assume the tools take care of everything. Another moral is that following instructions in full may be the wise initial path - the author probably detailed those steps for good reason.