How to fix the nfsfind "find: cannot open /path/to/directory: No such file or directory" error on OpenIndiana/Illumos
Tags: openindiana, illumos, nfsfind, unix
Published: 2021-05-10 05:18:32

My OpenIndiana machine recently emailed me the following error message, with the subject line `Cron <root@DellOptiPlex390MT> [ -x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind`:

```
find: cannot open /znapzend/DellOptiPlex390MT/ROOT/openindiana: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/export/home/jdrch: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/export: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/export/home: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/ROOT: No such file or directory
```

# Here's how I fixed it

1st, let's interpret the error message. Each line is saying that the `find` command cannot open a certain path, because there is no file or directory at that path. Both those paths exist, so how can `find` not locate them? More on that later.

The `Cron <root@DellOptiPlex390MT>`email subject line tells us that this error message is coming from the root crontab (or, more accurately, the `cron` daemon running as root), and that it's occurring at the `-x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind` line.

Let's [look at the root crontab](https://github.com/jdrch/Hardware/wiki/Useful-Links#crontab-1) to see if we can find a matching line. Sure enough, there it is:

```
15 3 * * 0 [ -x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind
```

The above line means "at 0315 every Sunday, run `-x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind`." So what is `nfsfind`? You can find the full description in the [Solaris docs](https://docs.oracle.com/cd/E37838_01/html/E61004/gqmry.html) (note that the `svc:/network/nfs/cleanup` and associated details do not apply to Illumos). `nfsfind` cleans stale temporary files out of your NFS shares once a week, presumably to prevent the shared filesystems from running out of space. 

Now that we know what `nfsfind` does, let's take a look at it using our editor of choice. I prefer `nano`, invoked here under my own user account as I do not want to accidentally edit a system script:

```
$ nano /usr/lib/fs/nfs/nfsfind
```

```
if [ ! -s /etc/dfs/sharetab ]; then exit ; fi

# Get all NFS filesystems exported with read-write permission.

DIRS=`/usr/bin/nawk '($3 != "nfs") { next }
        ($4 ~ /^rw$|^rw,|^rw=|,rw,|,rw=|,rw$/) { print $1; next }
        ($4 !~ /^ro$|^ro,|^ro=|,ro,|,ro=|,ro$/) { print $1 }' /etc/dfs/sharetab`

for dir in $DIRS
do
        find $dir -type f -name .nfs\* -mtime +7 -mount -exec rm -f {} \;
done
```

The penultimate line of the `nfsfind` script has the script's only `find` command. By process of elimination, this would be where the error message is coming from. It's safe to assume `find` isn't malfunctioning and its options are syntactically correct, so the error message is probably showing up because it's being fed the wrong input (`$dir`). 

`find $dir` tells us the `find` command is operating on a variable `$dir`, which from the `for dir in $DIRS` line is each successive value in `$DIRS`. From the `DIRS=` line we see that `DIRS` comes from whatever is found in `/etc/dfs/sharetab`.

Let's look at `/etc/dfs/sharetab`, invoking `nano` with the same privileges as before:

```
$ nano /etc/dfs/sharetab
```

```
/znapzend/DellOptiPlex390MT/ROOT/openindiana    -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/export/home/jdrch   -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT     -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/rpool1 -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend       -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/export      -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/export/home -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/ROOT        -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32

```

Now, some background:

* `/znapzend` is the mountpoint for `rpool1/znapzend/`, a ZFS fiesystem I [created](https://dev.to/jdrch/how-to-setup-znapzend-with-local-backup-on-openindiana-1f0p) as a destination for `znapzend`
* `rpool1` itself is mounted at `/rpool1`. I shared it via NFS using `# zfs set sharenfs=on` long before I created `rpool1/znapzend`

Clearly, all of `rpool1`'s child datasets inherited its `sharenfs=on` property upon their creation. Besides that, I had mounted `rpool1/znapzend` at `/znapzend`, which resulted in all `rpool1/znapzend`'s child datasets being mounted as described below:

```
$ mount | grep znapzend
/znapzend on rpool1/znapzend read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10008 on Fri Apr 30 21:59:04 2021
/znapzend/DellOptiPlex390MT on rpool1/znapzend/DellOptiPlex390MT read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10034 on Sat May  1 10:00:03 2021
/znapzend/DellOptiPlex390MT/ROOT on rpool1/znapzend/DellOptiPlex390MT/ROOT read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c1003a on Sat May  1 10:00:04 2021
/znapzend/DellOptiPlex390MT/ROOT/openindiana on rpool1/znapzend/DellOptiPlex390MT/ROOT/openindiana read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c1003c on Sat May  1 10:03:00 2021
/znapzend/DellOptiPlex390MT/export on rpool1/znapzend/DellOptiPlex390MT/export read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c1003d on Sat May  1 10:03:22 2021
/znapzend/DellOptiPlex390MT/export/home on rpool1/znapzend/DellOptiPlex390MT/export/home read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10040 on Sat May  1 10:03:33 2021
/znapzend/DellOptiPlex390MT/export/home/jdrch on rpool1/znapzend/DellOptiPlex390MT/export/home/jdrch read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10042 on Sat May  1 10:03:48 2021

```

It seems I made 2 mistakes here:

1. I forgot to `# zfs set sharenfs=off` the child datasets
2. I probably unnecessarily mounted the child datasets. As you can see from my `znapzend` [tutorial](https://dev.to/jdrch/how-to-setup-znapzend-with-local-backup-on-openindiana-1f0p), `znapzend` uses ZFS `zpool/dataset` paths, not filesystem paths (created by `mount` operations)

(As a corollary, this is probably why ZFS uses the term dataset and not filesystem. All ZFS filesystems are datasets, but not all ZFS datasets are filesystems. A dataset becomes a filesystem only when it is mounted.)

But that still doesn't explain why `find` chokes on those paths. Let's try to navigate to them ourselves using `cd`:

```
# cd /znapzend/DellOptiPlex390MT/ROOT/openindiana
-bash: cd: /znapzend/DellOptiPlex390MT/ROOT/openindiana: No such file or directory
```

Wait, what? How can there be no file or directory at that path? The answer lies in the sequence of sequence of events that led to that location being considered a *filesystem* (note the emphasis) path to begin with. 1st, `rpool1` was created with `sharenfs=on`. Much later `rpool1/znapzend` and `rpool1/znapzend/DellOptiPlex390MT` were created. Both those datasets inherited the `sharenfs=on`.

`rpool1/znapzend` was then mounted at `/znapzend`, which also mounted all of its current and future datasets. All of the above became filesystems by virtue of being mounted and NFS shares by virtue of inheriting their parent *dataset(s)'* `sharenfs=on` setting.

The future datasets came into being when `znapzend` created them recursively as `zfs receive` destinations. However, because each dataset is actually a family of snapshots, each has no actual corresponding filesystem, despite the apparent path! This is why `# cd` fails to find anything.

We can fix this problem by 1st unsharing the "problematic" dataset (actually this alone is sufficient to solve the problem):

```
# zfs set sharenfs=off rpool1/znapzend
```

and then also unmounting it (good practice, since the location isn't intended to be generally accessible to non-ZFS operations anyway):

```
# zfs unmount rpool1/znapzend
```

For those who may be confused about the continued accessibility of the destination dataset to `znapzend` after `unmount`, remember that all datasets on a zpool are accessible via *ZFS* (note the emphasis) commands as long as that zpool has been imported (automatic on OpenIndiana and FreeBSD for zpools created on the same machine) and has not been exported.

Checking the contents of `/etc/dfs/sharetab` again:

```
$ nano /etc/dfs/sharetab
```

gives:

```
/rpool1 -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
```

Both `find` and 'cd` work on `/rpool1`, so we can be sure there will be no further errors of the kind detailed at the outset.

----

Thanks to Joshua M. Clulow & Alan Coopersmith on the openindiana-discuss mailing list for helping me resolve this problem.