Storage on Kubernetes
Tags: kubernetes, storage, linux
Published: 2019-07-28 10:50:38

---
title: Storage on Kubernetes
published: true
description: An overview of Kubernetes storage and Rook
tags: Kubernetes, Storage, Linux
---


Kubernetes is fantastic for stateless apps. Deploy your app, scale it up to hundreds of instances, be happy. But how do you manage **storage**? How do we ensure every one of our hundreds of apps gets a piece of reliable, fast, cheap storage?


Let's take a typical Kubernetes cluster, with a handful of nodes (Linux servers) powering a few replicas of our app:

![storage](https://kubesail.com/blog-images/blog-storage-rook-2.jpg)

Notice our sad, unused disk drives! Kubernetes sure brings a lot of wins, but are we even sysadmins anymore if we don't manage enormous RAID arrays?

### Persistent Volume Claims

In Kubernetes, we define **PersistentVolumeClaims** to ask our system for storage. To put it simply, an App "claims" a bit of storage, and the system responds in a configurable way:

<img style="width: 400px" src="https://kubesail.com/blog-images/blog-storage-0.jpg" />

Unfortunately, most Cloud providers are eager to harness the simplicity of Kubernetes by "replying" to your storage request by attaching Cloud Storage (eg: Amazon's EBS).

<img style="width: 400px" src="https://kubesail.com/blog-images/blog-storage-1.jpg" />

There are a number of downsides to this strategy for consumers:

  - **Performance & Cost**: EBS Volumes (and other cloud provider's equivilent) performance depends on its size - meaning a *smaller disk* is the same thing as a *slower disk*.
  - **Characteristics**: Cloud Providers typically offer HDD, SDD, and a "provisioned IO" option. This limits sysadmins in their storage system designs - Where is the tape backup? What about NVMe? How is the disk attached to the server my code is running on?
  - **Portability / Lock-in**: EBS is EBS and Google Persistent Disks are Google Persistent Disks. Cloud vendors are aggressively trying to lock you into their platforms - and typically hide the filesystem tools we know and love behind a Cloud-specific snapshotting system.

So... What we need then is an **Open Source** storage orchestration system, which would run on any Kubernetes cluster, and can transform piles of drives into pools of storage, available to our Pods:

<img style="width: 600px" src="https://kubesail.com/blog-images/blog-storage-2.jpg" />

### Enter Rook.io

What is **Rook**? From the website, Rook is "an Open-Source, Cloud-Native Storage for Kubernetes" with "Production ready File, Block and Object Storage". Marketing speak aside, Rook is an open-source version of AWS EBS and S3, which you can install on your own clusters. It's also the backend for [KubeSail](https://kubesail.com)'s storage system, and it's how we carve up massive RAID arrays to power our users apps!

### Rook and Ceph

Rook is a system which sits in your cluster and responds to requests for storage, but it is not itself a storage system. Although this makes things a bit more complex, it should make you feel __good__, because while Rook is quite new, the storage system it uses under the hood are battle hardened, and _far from beta_.

We'll be using [**Ceph**](https://ceph.com/), which is [about 15 years old](https://en.wikipedia.org/wiki/Ceph_(software)#History) and is in use and developed by companies like Canonical, CERN, Cisco, Fujitsu, Intel, Red Hat, and SanDisk. It's very far from Kubernetes-hipster, despite how cool the Rook project looks!

With **Rook** at version 1.0, and **Ceph** powering some of the world's most important datasets, I'd say it's about time we get confident and take back control of our data-storage! I'll be building RAID arrays in the 2020s and no one can stop me! *Muahaha*

### Install

I won't focus on initial install here, since [the Rook guide is quite nice](https://rook.io/docs/rook/v1.0/ceph-quickstart.html). If you want to learn more about the Rook project, I recommend [this KubeCon video](https://www.youtube.com/watch?v=pwVsFHy2EdE) as well. Once you've got Rook installed, we'll create a few components:

- A **CephCluster**, which maps nodes and their disks to our new storage system.
- A **CephBlockPool**, which defines how to store data, including how many replicas we want.
- A **StorageClass**, which defines a way of using storage from the CephBlockPool.

Let's start with the **CephCluster**:
```yaml
#?filename=ceph-cluster.yaml&mini=true&noApply=true
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  mon:
    count: 3
    allowMultiplePerNode: false
  storage:
    useAllNodes: false
    useAllDevices: true
    location:
    config:
      osdsPerDevice: "1"
    directories:
      - path: /opt/rook
    nodes:
    - name: "my-storage-node-1"
    - name: "my-storage-node-2"
    - name: "my-storage-node-3"
```

The most important things to note here is that a **CephCluster** is fundamentally a map of which **Nodes** and which drives or directories on those Nodes will be used to store data. Many tutorials will suggest `useAllNodes: true`, which we strongly recommend against. Instead, we recommend managing a smaller subset pool of "storage workers" - this allows you to use different system types (with say, very slow drives) later without accidentally/unknowingly adding it to the storage pool. We'll be assuming `/opt/rook` is a mount-point, but Rook is capable of using unformatted disks as well as directories.

One other note is that a `mon` is Rook's monitoring system. We strongly suggest running at least three and ensuring `allowMultiplePerNode` is false.

Now our cluster looks like this:

![storage](https://kubesail.com/blog-images/blog-storage-rook-3.jpg)

By the way, you'll want to take a look at the pods running int the `rook-ceph` namespace! You'll find your OSD (Object-Storage-Device) pods, as well as monitoring and agent pods living in that namespace. Let's create our **CephBlockPool** and a **StorageClass** that uses it:

```yaml
#?filename=ceph-blockpool.yaml&mini=true&noApply=true
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: my-storage-pool
  namespace: rook-ceph
spec:
  failureDomain: osd
  replicated:
    size: 2
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: my-storage
provisioner: rook.io/block
parameters:
  pool: my-storage-pool
```

Note that the replicated size is set to 2, meaning we'll have 2 copies of all our data.

We'll wait for our CephCluster to get settled - you'll want to take a look at the `CephCluster` object you created:

```
➜ kubectl -n rook-ceph get cephcluster rook-ceph -o json | jq .status.ceph.health
"HEALTH_OK"
```

Now, we're ready to make a request for storage!

![storage](https://kubesail.com/blog-images/blog-storage-rook-4.jpg)

We can request storage in a number of standard ways now, from this point there is zero "Rook-specific" code or assumptions:

```yaml
#?filename=ceph-pvc.yaml&mini=true&noApply=true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-data
spec:
  accessModes:
    - ReadWriteOnce
  # Use our new ceph storage:
  storageClassName: my-storage
  resources:
    requests:
      storage: 1000Mi
```

Now Rook will see our **PersistentVolumeClaim** and create a **PersistentVolume** for us! Let's take a look:

```
➜ kubectl get pv --watch
NAME     CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM              STORAGECLASS    AGE
pvc-...  1000Mi    RWO           Delete          Bound   default/test-data  my-storage      13m
```

So there we go! Successful, fairly easy to use, Kubernetes-native storage system. We can bring our own disks, we can use any cloud provider... Freedom!

![storage](https://kubesail.com/blog-images/blog-storage-rook-6.jpg)

### Conclusion: Should you use it?

Well, TLDR: Yes.

But the real question you should ask is, am I trying to build the most awesome data-center-in-my-closet the world has seen since 2005? Am I trying to learn enterprise-ready DevOps skills? If the answer is yes, I strongly recommend playing with Rook.

At KubeSail we use Rook internally, and offer a `standard` storage class which provides extremely fast NVMe storage - even on our *free* tier! We manage multiple clusters with extremely large and busy Ceph clusters, and so far have been extremely happy in terms of stability and performance. While playing with piles of harddrives does make me nostalgic for the good ol' days, our platform allows you to have a rock-solid setup for your apps out of the box.

We strongly prefer Open Source tools that avoid vendor lock-in. You should be able to install and configure all of our tools at KubeSail on your own clusters, and we'll continue to write about our stack in a continuing series of blog posts. However, if all you want is super fast, super cheap storage attached to your Docker containers on multiple servers, checkout our Hosted Platform at [KubeSail.com](https://kubesail.com)!

If you have any questions about the article, Rook, Kubernetes, or really just about anything else, send us a message [on Twitter](https://twitter.com/kubesail)!