Leveraging LVM Caching
Tags: lvm, linux
Published: 2019-02-27 16:22:04

---
title: Leveraging LVM Caching
published: true
description: 
tags: lvm, linux
canonical_url: https://mrkc.me/blog/leveraging-lvm-caching.html
---

Note: I'm migrating relevant blog posts I've written over to dev.to. This is the first one.

---

I recently implemented [LVM caching](http://man7.org/linux/man-pages/man7/lvmcache.7.html) based largely on [Richard WM Jones' post](https://rwmj.wordpress.com/2014/05/22/using-lvms-new-cache-feature/) and thought I'd document my own experience here.

I started with an SSD containing `/`, `/home`, and `swap`, two 2 TB HDDs (`sdb`, and `sdc`) and an additional SSD (`sdd`).

```
[root@bender ~]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0 447.1G  0 disk
├─sda1            8:1    0   500M  0 part /boot
└─sda2            8:2    0 446.7G  0 part
  ├─fedora-root 253:0    0    50G  0 lvm  /
  ├─fedora-swap 253:1    0  15.7G  0 lvm  [SWAP]
  └─fedora-home 253:2    0   381G  0 lvm  /home
sdb               8:16   0   1.8T  0 disk
sdc               8:32   0   1.8T  0 disk
sdd               8:48   0 119.2G  0 disk
```

I then used `fdisk` to create a `Linux` partition on each HDD and a `Linux LVM` partition on the SSD. Though, these partition types don't actually matter much for our purposes beyond looking nice.

```
[root@bender ~]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0 447.1G  0 disk
├─sda1            8:1    0   500M  0 part /boot
└─sda2            8:2    0 446.7G  0 part
  ├─fedora-root 253:0    0    50G  0 lvm  /
  ├─fedora-swap 253:1    0  15.7G  0 lvm  [SWAP]
  └─fedora-home 253:2    0   381G  0 lvm  /home
sdb               8:16   0   1.8T  0 disk
└─sdb1            8:17   0   1.8T  0 part
sdc               8:32   0   1.8T  0 disk
└─sdc1            8:33   0   1.8T  0 part
sdd               8:48   0 119.2G  0 disk
└─sdd1            8:49   0 119.2G  0 part
```

I then created a RAID 1 set using the two HDD partitions.

```
[root@bender ~]# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@bender ~]# echo 'DEVICE /dev/sdb1 /dev/sdc1' > /etc/mdadm.conf
[root@bender ~]# mdadm --detail --scan >> /etc/mdadm.conf
[root@bender ~]# cat /etc/mdadm.conf
DEVICE /dev/sdb1 /dev/sdc1
ARRAY /dev/md0 metadata=1.2 name=bender:0 UUID=2f9fd7a2:cf0d4dd8:579ec7ce:073d0886
```

Then I created the LVM volumes and an `ext4` filesystem on the RAIDed volume.

```
[root@bender ~]# pvcreate /dev/md0 /dev/sdd1
  Physical volume "/dev/md0" successfully created
  Physical volume "/dev/sdd1" successfully created
[root@bender ~]# vgcreate data /dev/md0
  Volume group "data" successfully created
[root@bender ~]# lvcreate -n vol0 -L 1.5T data /dev/md0
  Logical volume "vol0" created.
[root@bender ~]# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vol0 data   -wi-a-----   1.50t
  home fedora -wi-ao---- 380.95g
  root fedora -wi-ao----  50.00g
  swap fedora -wi-ao----  15.69g
[root@bender ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/md0   data   lvm2 a--    1.82t 326.89g
  /dev/sda2  fedora lvm2 a--  446.64g   4.00m
  /dev/sdd1         lvm2 ---  119.24g 119.24g
[root@bender ~]# mkfs.ext4 /dev/data/vol0
mke2fs 1.42.13 (17-May-2015)
Creating filesystem with 402653184 4k blocks and 100663296 inodes
Filesystem UUID: 51e8796a-5090-45ce-ae62-4944f2eb9132
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
```

Now for the caching configuration. It's important to specify the actual partition (`sdd1`) in the following commands to ensure that the cache gets created on that device and not arbitrarily within the volume group.

```
[root@bender ~]# vgextend data /dev/sdd1
  Volume group "data" successfully extended
[root@bender ~]# lvcreate -L 1G -n vol0_cache_meta data /dev/sdd1
  Logical volume "vol0_cache_meta" created.
[root@bender ~]# lvcreate -L 115G -n vol0_cache data /dev/sdd1
  Logical volume "vol0_cache" created.
[root@bender ~]# lvs
  LV              VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vol0            data   -wi-a-----   1.50t
  vol0_cache      data   -wi-a----- 115.00g
  vol0_cache_meta data   -wi-a-----   1.00g
  home            fedora -wi-ao---- 380.95g
  root            fedora -wi-ao----  50.00g
  swap            fedora -wi-ao----  15.69g
[root@bender ~]# lvconvert --type cache-pool --poolmetadata data/vol0_cache_meta data/vol0_cache
  WARNING: Converting logical volume data/vol0_cache and data/vol0_cache_meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert data/vol0_cache and data/vol0_cache_meta? [y/n]: y
  Converted data/vol0_cache to cache pool.
[root@bender ~]# lvconvert --type cache --cachepool data/vol0_cache data/vol0
  Logical volume data/vol0 is now cached.
[root@bender ~]# lvs
  LV   VG     Attr       LSize   Pool         Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  vol0 data   Cwi-a-C---   1.50t [vol0_cache] [vol0_corig] 0.00   1.43            100.00
  home fedora -wi-ao---- 380.95g
  root fedora -wi-ao----  50.00g
  swap fedora -wi-ao----  15.69g
```

Then I mounted it at `/var/data`.

```
[root@bender ~]# echo '/dev/mapper/data-vol0 /var/data                 ext4    defaults        1 2' >> /etc/fstab
[root@bender ~]# mount -a
[root@bender ~]# df -h /var/data
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/data-vol0  1.5T   69M  1.5T   1% /var/data
```