Home dir automation with Ansible PowerScale / Isilon
Tags: ansible, dell, isilon, devops
Published: 2020-09-04 09:55:10

---
title: Home dir automation with Ansible PowerScale / Isilon
published: true
date: 2020-08-29 07:00:00 UTC
tags: #ansible #dell #isilon #devops
canonical_url: http://0.0.0.0:4321/ansible-isilon.html
---

# TL;DR

[ansible-isilon](https://github.com/dell/ansible-isilon) eases the admin tasks on Isilon / PowerScale ; watch how cool it can be on [Youtube](https://www.youtube.com/watch?v=RF5WoeRry1k&list=PLbssOJyyvHuVXyKi0c9Z7NLqBiDiwF1eA&index=3) and how to use it below.OverlayFS is great but has some limitations for some use-cases ; [UnionFS](https://en%0A.wikipedia.org/wiki/UnionFS) is not dead !

# Table of Contents

- [The premise](#the-premise)
- [The implementation](#the-implementation)
  - [Install Ansible modules for PowerScale/Isilon](#install-ansible-modules-for-powerscaleisilon)
  - [The files](#the-files)
  - [List usage in Ansible](#list-usage-in-ansible)
  - [UnionFS](#unionfs)
  - [File system removal](#file-system-removal)
- [Video](#video)

# The premise

In my old days at the university, I used to work [Sun Ray](https://en.wikipedia.org/wiki/Sun_Ray) thin client (imagine the evolution between [VT100](https://en.wikipedia.org/wiki/VT100) to [modern VDI](https://www.delltechnologies.com/en-us/solutions/vdi/index.htm)). Students and teachers were all connected to the same SPARC server to work. Each of us had its own home directory accessible from the NFS server.

More than 15 years later, enterprises of any size still use home directories on NFS for their users !

In the following article, we will show how to use [Ansible](https://www.ansible.com/) to manage home directories hosted on a PowerScale array in a university.The predicate is that [Active Directory](https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/manage/ad-ds-simplified-administration) **is the reference** for the userbase. Each LDAP user can be either in the _student_ group or the _teacher_ group.Any student or teacher in AD must have his homedir in PowerScale and be accessible via NFS exports. Any student who is no longer enrolled and not in AD will have thier homedir removed.

The ansible playbook will :

- get the list of students and teachers from AD
- create a unix home directory in PowerScale/Isilon for each user
- set different quotas if the user is a student or a teacher
- have daily snapshots of the home directories with varying policies of retention if for the students and teachers
- mount the home directories in a list of Unix server
- cleanup the home directories of students that are not in the AD anymore

# The implementation

In this chapter I will not detail all the tasks as most of them are self-explanatory, but, describe a few tips & tricks that can be reused in other playbooks.

## Install Ansible modules for PowerScale/Isilon

The [Product Guide](https://github.com/dell/ansible-isilon/blob/master/dellemc_ansible/docs/Ansible%20for%20Dell%20EMC%20Isilon%20v1.1%20Product%20Guide.pdf) documents the module installation and usage (equivalent to `ansible-doc dellemc_isilon_[module]`).

This example comes with a [Dockerfile](https://github.com/dell/ansible-storage-automation/blob/powerscale/Dockerfile) that has the required dependencies to run the playbook.

As the [ansible-isilon](https://github.com/dell/ansible-isilon) is very specific about Isilon SDK version, the most important line is :

```
RUN pip3 install isi-sdk-8-1-1 pywinrm && \
    git clone https://github.com/dell/ansible-isilon.git

```

Once `docker build`-ed, you can execute the playbook with it with :

```
podman run --security-opt label=disable -e ANSIBLE_HOST_KEY_CHECKING=False \
           -v ~/.ssh/id_rsa.emc.pub:/root/.ssh/id_rsa.pub -v ~/.ssh/id_rsa.emc:/root/.ssh/id_rsa \
           -v "$(pwd)"/homedir/:/ansible-isilon \
           -ti docker.io/coulof/ansible-isilon:1.1.0 ansible-playbook \
           -i /ansible-isilon/hosts.ini /ansible-isilon/create_homedir_for_ad_users_in_isilon.yml

```

Note that on my Fedora 32 machine, the `--security-opt label=disable` is mandatory to be able to mount the volumes.

## The files

To use the playbook, you will have to update a couple of files:

- [hosts.ini](https://github.com/dell/ansible-storage-automation/blob/homedir/powerscale/homedir/hosts.ini) ; which has the inventory of Unix and Domain Controller
- [credentials-isi.yml](https://github.com/dell/ansible-storage-automation/blob/homedir/powerscale/homedir/credentials-isi.yml) ; which has the details of the PowerScale
- [create\_homedir\_for\_ad\_users\_in\_isilon.yml](https://github.com/dell/ansible-storage-automation/blob/homedir/powerscale/homedir/create_homedir_for_ad_users_in_isilon.yml) ; which is the playbook with all the tasks, where you have to update the variables `base_path` and `nfs_server_ip` in several sections to point to the PowerScale path and IP.

## List usage in Ansible

The first tip is in task [Get userbase from Active Directory](https://github.com/dell/ansible-storage-automation/blob/powerscale/homedir/create_homedir_for_ad_users_in_isilon.yml#L2) with :

```
    - set_fact:
        students_list: "{{members_students_group.members | list}}"
        teachers_list: "{{members_teachers_group.members | list}}"

```

The `set_fact` creates two lists of users that will be reused across the playbook.With the object list, we can loop through and execute the same task for each user as done in the [FS creation](https://github.com/dell/ansible-storage-automation/blob/powerscale/homedir/create_homedir_for_ad_users_in_isilon.yml#L49) :

```
      dellemc_isilon_filesystem:
        <<: *isi_connection_vars
        path: "{{base_path}}/students/{{item}}"
        ...
        state: 'present'
      loop: "{{ hostvars['devconad.com']['students_list'] }}"

```

Or make it easy to find orphan homedirs by playing with list operations when [listing unix mounted dirs](https://github.com/dell/ansible-storage-automation/blob/homedir/powerscale/homedir/create_homedir_for_ad_users_in_isilon.yml#L167) :

```
    - name: Capture files in path and register
      shell: >
        ls -1 /mnt/nfs_students
      register: students_home_dir
      run_once: True
    - set_fact:
        orphan_home_dirs: "{{students_home_dir.stdout_lines | list | difference(hostvars['devconad.com']['students_list'])}}"

```

## UnionFS

To stick with the usual `/home/<username>` file system hierarchy, I wanted to mount the students and teachers sub-dirs within the same `/home` and keep the write in the lower dirs as follow :

```
/mnt/nfs_teachers/ /mnt/nfs_students/ /home
├── alice ├── carol ├── alice
└── bob └── dan ├── bob
                                                ├── carol
                                                └── dan

```

The capability of writing in lowerdirs live is available in [AuFS](https://en.wikipedia.org/wiki/Aufs) and [UnionFS](https://en.wikipedia.org/wiki/UnionFS) but not in the very popular [OverlayFS](https://en.wikipedia.org/wiki/OverlayFS).

As stated by the [Kernel documentation](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt):

> Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed.

There are plenty of discussions about that topic on [Stackoverflow](https://unix.stackexchange.com/questions/393930/merge-changes-to-upper-filesystem-to-lower-filesystem-in-linux-overlay-overlayf).

To achieve it I used `unionfs-fuse` which is available from Ubuntu repo or [CentOS third-party repo](https://centos.pkgs.org/7/repoforge-x86_64/fuse-unionfs-0.26-1.el7.rf.x86_64.rpm.html). The obvious advantage of Filesystem in Userspace is that I won’t need to recompile the Linux kernel to use it. In the [/etc/fstab](https://github.com/dell/ansible-storage-automation/blob/homedir/powerscale/homedir/create_homedir_for_ad_users_in_isilon.yml#L154) we can use `unionfs#` to mount a FUSE filesystem :

```
        line: "unionfs#/mnt/nfs_students=RW:/mnt/nfs_teachers=RW /home/ fuse cow 0 0"

```

## File system removal

It is possible to [remove PowerScale/Isilon file system](https://github.com/dell/ansible-storage-automation/blob/homedir/powerscale/homedir/create_homedir_for_ad_users_in_isilon.yml#L182) with the Ansible directive :

```
    - name: Remove Filesystem and Quota for missing students from AD
      dellemc_isilon_filesystem:
        path: "{{base_path}}/students/{{item}}"
        quota:
          quota_state: absent
        state: absent

```

Note that by design, the Ansible module will only remove the directory **if empty**.If you need to remove a non-empty directory, you have to issue REST call directly.

# Video

For a live demo, check the video here: https://www.youtube.com/watch?v=RF5WoeRry1k&list=PLbssOJyyvHuVXyKi0c9Z7NLqBiDiwF1eA&index=2