K8s mount PV with SELinux
Published: 2020-07-22 14:59:27

---
title: K8s mount PV with SELinux
published: true
date: 2020-07-14 07:00:00 UTC
tags: 
canonical_url: https://storage-chaos.io/k8s-selinux-mount-pv.html
---

# TL; DR

If you want to use Kubernetes with [SELinux](https://wiki.gentoo.org/wiki/SELinux) and mount `PersistentVolume`, you have to make sure your mounted FS has labels. You can do it with the [mountOptions](https://unofficial-kubernetes.readthedocs.io/en/latest/concepts/storage/persistent-volumes/#mount-options) `-o context="system_u:object_r:container_var_lib_t:s0"` and if your driver doesn’t support it, you can write an SELinux policy like [this one](https://gist.github.com/coulof/9df7c9f3178ecf6706b0c5316ab9de7e).

# The premise

The problem came from a customer who is using [Docker Enterprise](https://www.mirantis.com/software/docker/docker-enterprise/) with [CSI Driver for VxFlexOS](https://github.com/dell/csi-vxflexos/) (now [PowerFlex](https://www.delltechnologies.com/en-us/storage/powerflex.htm)), and SELinux enforced on the nodes.

Anytime a Pod tried to write data on the `PersistentVolume`, we had `Permission denied` error from the OS.

SELinux relies on file label (sometimes call context) in the file extended attributes to apply its policies. For examples below :

```
ls -lZd /root /home /etc
drwxr-xr-x. root root system_u:object_r:etc_t:s0 /etc
drwxr-xr-x. root root system_u:object_r:home_root_t:s0 /home
dr-xr-x---. root root system_u:object_r:admin_home_t:s0 /root

```

By default, on a newly formatted and mounted FS the files are unlabeled :

```
mkfs.ext4 /dev/loop0
mount -o loop /dev/loop0 /media/xxx
ls -Z
drwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 xxx

```

On the customer setup, it was forbidden to do any action on an unlabeled file.

# How to get the new FS labeled ?

The typical way to relabel a directory is to use the command `restorecon` against the mount point, which will restore the security context and label.Because we are in a containerized environment with volumes dynamically mounted by Kubernetes, this is not realistic to expect the `restorecon` command to be executed each time a volume is mounted.

Another option, especially suitable to big FileSystems with many files, is to can create a file named `.autorelabel` at the root level, so it forces a relabel on the mountpoint. Slightly better but, still, not feasible in a dynamic environment like ours.

A better option is to mount the FS with the [context option](https://www.man7.org/linux/man-pages/man8/mount.8.html#FILESYSTEM-INDEPENDENT_MOUNT_OPTIONS). That option is my favorite ❣

Unfortunately, the DellEMC CSI drivers don’t have the `mountOption` capability at the time of that post. That feature is on the roadmap, but in the meantime, I needed a plan B.

The last possibility and the one we implemented is to write a specific policy to allow containers to manipulate unlabeled files and directories.

Since SELinux follows a model of the least-privilege (aka you can’t do anything except if explicitly allowlisted), the challenge was to have all the syscalls a container needs to do their job.

To get inspiration, I hijacked policies given in the [SELinux Project](https://github.com/SELinuxProject/refpolicy/tree/master/policy/modules/apps) and came up with that list : `class file { create open getattr setattr read write append rename link unlink ioctl lock };`

The full policy is :
```c
module vxflexos-cni 1.0;

require {
	type unlabeled_t;
	type container_t;
	class file { create open getattr setattr read write append rename link unlink ioctl lock };
	class dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink write };
}

#!!!! WARNING: 'unlabeled_t' is a base type.
allow container_t unlabeled_t:dir { create open getattr setattr read write link unlink rename search add_name remove_name reparent rmdir lock ioctl };
allow container_t unlabeled_t:file { create open getattr setattr read write append rename link unlink ioctl lock };

```
To compile it, you will need the SELinux Devel package (in Fedora : `dnf install selinux-policy-devel.noarch` will do) and then compile the policy with : `make -f /usr/share/selinux/devel/Makefile`. To install the newly compiled policy run : `semodule -i vxflexos-cni.pp`.

# Wrap-up

`mountOptions` capability is coming with every Dell Technologies CSI driver in the incoming months.

The Gentoo website is a gold mine of information for SELinux ! To understand better the issue, I mostly read these pages on [Labels](https://wiki.gentoo.org/wiki/SELinux/Labels), the [Tutorial to create a policy](https://wiki.gentoo.org/wiki/SELinux/Tutorials/Creating_your_own_policy_module_file), and [SELinux Project Policies](https://github.com/SELinuxProject/refpolicy/tree/master/policy/modules/apps).