Mocking the filesystem in Go using Afero
Tags: go, afero, testing, mocking
Published: 2020-12-16 17:21:06

Let's start the project

```bash
cd $GOPATH/src/github.com/drpaneas
mkdir checkfile; cd checkfile
git init -q
go mod init
touch main.go
code . # for vscode
```

## Coding Task

Write a function that checks if a specific directory exists on disk. _For example_:

```go
if folderExists(directory) {
  // do something
}
```

Bonus: Feel free to add `fileExists(filename)` as homework.

### Acceptance Criteria

- Use https://github.com/spf13/afero for filesystem abstraction
- Make sure it works cross-platform
- For logging use https://github.com/Sirupsen/logrus
- Put the function into `pkg/utils` package
- Write a unit test
- Make sure there are [testable examples](https://blog.golang.org/examples)

**Hints**:

- Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself. The last resort is mocking the filesystem. There are quite a few powerful libraries like spf13/afero for this purpose (mocking of a filesystem). These packages will create temporary files in the background and clean up afterward.
- As an exercise, try to write the test first. Let it fail. Then try to fix it, by writing the actual implementation of the function.

### Extra Help

- https://golangcode.com/check-if-a-file-exists/
- https://www.youtube.com/watch?v=ifBUfIb7kdo
- https://github.com/spf13/afero#using-afero-for-testing
- An example using Afero for testing: http://krsacme.com/golang-fs-tests/
- For logging https://linuxhint.com/golang-logrus-package/

**NOTICE**: 

If you are unable to mock this test, in worst case scenario, create a folder `pkg/utils/testdata` and put actual files you can use for your test.

## Let's code

```go
// main.go
package main

import (
	"os"

	log "github.com/sirupsen/logrus"
)

// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
	if path == "" {
		log.Debug("Path is empty")
		return false
	}

	info, err := os.Stat(path)
	if err != nil {
		if os.IsNotExist(err) { // look for the specific error type
			return false
		}
	}

	return isDir == info.IsDir()
}

// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
	return exists(path, true)
}

// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
	return exists(path, false)
}


func main() {
	// Just for demo purposes
	if FolderExists("logs") {
		log.Info("Folder exists")
	} else {
		log.Error("Folder does not exist")
	}

	if FileExists("helloworld.txt") {
		log.Info("File exists")
	} else {
		log.Error("File does not exist")
	}
}
```

One way to test this is by actually using the program itself.
Running it as it is, it's expected to fail:

```shell
ERRO[0000] Folder does not exist                        
ERRO[0000] Folder does not exist
```

If we create the two necessary things we look for, then it's expected to work:

```bash
# Create the file and directory
touch helloworld.txt
mkdir logs

# Run it again
go run main.go
INFO[0000] Folder exists                                
INFO[0000] File exists
```

Normally you would create separate package and not use `main`.
So let's do it!

```shell
mkdir -p pkg/utils/
touch pkg/utils/filesystem.go
```

The code will look like this:

```go
// main.go
// -------

package main

import (
	"github.com/drpaneas/checkfile/pkg/utils"
	log "github.com/sirupsen/logrus"
)

func main() {
	if utils.FolderExists("logs") {
		log.Info("Folder exists")
	} else {
		log.Error("Folder does not exist")
	}

	if utils.FileExists("helloworld.txt") {
		log.Info("File exists")
	} else {
		log.Error("File does not exist")
	}
}
```

```go
// pkg/utils/filesystem.go
// -----------------------

package utils

import (
	"os"

	log "github.com/sirupsen/logrus"
)

// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
	if path == "" {
		log.Debug("Path is empty")
		return false
	}

	info, err := os.Stat(path)
	if err != nil {
		if os.IsNotExist(err) { // look for the specific error type
			return false
		}
	}

	return isDir == info.IsDir()
}

// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
	return exists(path, true)
}

// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
	return exists(path, false)
}
```

Just to be on the safe side we didn't forget anything major, let's run it again to make sure everything is alright.

```bash
# Run it again
go run main.go
INFO[0000] Folder exists                                
INFO[0000] File exists
```

## Test with testdata

Ok,we need to write tests for `pkg/utils/filesystem.go`.
One way of doing this is by accessing a real file system to do testing, then we're talking about what you'd call system or integration testing. That's fine in itself: for instance, you could run such tests in a throw-away container as part of a CI pipeline.
But with this approach you can't sensibly do what is called unit-testing.

To proceed with that, we are going to use the generated boilerplate code that `vscode` happily offers (give you have the Go extensions installed).

Place the mouse cursor over the `FolderExists()` function and press the keys: `CTRL+SHIFT+P`.
From the drop-down menu, type: `Go: test` and select the one for the function.
This will create the `filesystem_test.go` file and populate it with a table test.
Add your test logic there:

```go
// pkg/utils/filesystem_test.go
// ----------------------------

package utils

import "testing"

// Add those variables as well
var (
	existingFolder    = "./testdata/a-folder-that-exists"
	nonExistingFolder = "./testdata/a-folder-that-does-not-exists"
)

func TestFolderExists(t *testing.T) {
	type args struct {
		path string
	}
	tests := []struct {
		name string
		args args
		want bool
	}{
		// ----- test logic start ----- //
		{
			"Returns true when given folder exists",
			args{path: existingFolder},
			true,
		},
		{
			"Returns false when given folder does not exist",
			args{path: nonExistingFolder},
			false,
		},
		// ----- test logic end ----- //
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if got := FolderExists(tt.args.path); got != tt.want {
				t.Errorf("FolderExists() = %v, want %v", got, tt.want)
			}
		})
	}
}
```

The tests is relying upon real data on the actual hard drive.
Run tests and you will see they will fail:

```go
go test ./... -v
?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
    filesystem_test.go:33: FolderExists() = false, want true
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- FAIL: TestFolderExists (0.00s)
    --- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
FAIL
FAIL    github.com/drpaneas/checkfile/pkg/utils 0.098s
FAIL
```

Create the necessary testdata and try again:

```shell
mkdir -p pkg/utils/testdata/a-folder-that-exists
```

```shell
$ go test ./... -v?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils (cached)
```
Now, see the coverage. Go into the package and run `cover`.

```shell
 cover
PASS
coverage: 77.8% of statements
ok      github.com/drpaneas/checkfile/pkg/utils 0.293s
```

PS: You need to have this snippet into your `~/.bashrc`:

```bash
# Go coverage trick
cover () {
    t="/tmp/go-cover.$$.tmp"
    go test -coverprofile=$t $@ && go tool cover -html=$t && unlink $t
}
```

Let's enhance the testing coverage, by adding:

```go
		{
			"Returns false when provided path is empty",
			args{path: ""},
			false,
		},
```

Run the tests again and the coverage:

```shell
go test ./...

$ cover ./pkg/utils
ok      github.com/drpaneas/checkfile/pkg/utils 0.232s  coverage: 88.9% of statements
```

Add the tests for the other function as well:

The variables first:

```go
var (
	existingFile      = "./testdata/a-file-that-exists"
	nonExistingFile   = "./testdata/a-file-that-doesn-not-exist"
)
```

The test code:

```go
		{
			"Returns true when given file exists",
			args{path: existingFile},
			true,
		},
		{
			"Returns false when given file does not exist",
			args{path: nonExistingFile},
			false,
		},
		{
			"Returns false when provided path is empty",
			args{path: ""},
			false,
		},
```

And create the appropriate file for testing:

```shell
touch pkg/utils/testdata/a-file-that-exists
```

Run the tests (they should all pass):

```shell
$ go test -v ./...?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
=== RUN   TestFileExists
=== RUN   TestFileExists/Returns_true_when_given_file_exists
=== RUN   TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN   TestFileExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFileExists (0.00s)
    --- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
    --- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils 0.301s
```

Check the testing coverage:

```shell
cover ./pkg/utils
ok      github.com/drpaneas/checkfile/pkg/utils 0.268s  coverage: 100.0% of statements
```

Although we already hit the 100% testing coverage, I would like to add another test-case.

Add these two code block respectively:


```go
// TestFolderExists()
// ...
		{
			"Returns false when provided path is not a directory",
			args{path: existingFile},
			false,
		},
```

```go
// TestFileExists
// ...
		{
			"Returns false when provided path is not a file",
			args{path: existingFolder},
			false,
		},
```

... and that's it!
Run again the tests to verify everything works and we are good to go.

```shell
go test ./...
?       github.com/drpaneas/checkfile   [no test files]
ok      github.com/drpaneas/checkfile/pkg/utils 0.332s
```

## Test without testdata

Ok now let's do the same without the actual testdata.
To unit-test your function you need to replace something it uses with something "virtualised". 
Exactly how to do that is an open question.
Right now, a work is being done on [bringing filesystem virtualisation](https://go.googlesource.com/proposal/+/master/design/draft-iofs.md) right into the Go standard library, but while it's not there yet, there's a 3rd-party package which has been doing that for ages, — for instance, github.com/spf13/afero.

First let's remove the actual real data, and see the tests fail:

```shell
rm -r ./pkg/utils/testdata
```

So now the tests are failing obviously:

```shell
go test ./...
?       github.com/drpaneas/checkfile   [no test files]
--- FAIL: TestFolderExists (0.00s)
    --- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
        filesystem_test.go:47: FolderExists() = false, want true
--- FAIL: TestFileExists (0.00s)
    --- FAIL: TestFileExists/Returns_true_when_given_file_exists (0.00s)
        filesystem_test.go:86: FileExists() = false, want true
FAIL
FAIL    github.com/drpaneas/checkfile/pkg/utils 0.404s
FAIL
```

To do that we will use [Afero](https://github.com/spf13/afero).
Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself. 

Let's modify our code to use Afero for **both implementation and test**.
So for the `filesystem.go` just add:

```go
// Making the global, sharing the memory filesystem
var (
	FS  afero.Fs     = afero.NewMemMapFs()
	AFS *afero.Afero = &afero.Afero{Fs: FS}
)
```

and replace the `os.Stat` to `AFS.Stat`. That's all.
For the `filesystem_test.go` just add:

```go
func init() {
	// create test files and directories
	AFS.MkdirAll(existingFolder, 0755)
	afero.WriteFile(AFS, existingFile, []byte("test file"), 0755)
}
```

Run the tests and they should pass:

```shell
go test ./... -v
?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_empty
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_not_a_directory
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_not_a_directory (0.00s)
=== RUN   TestFileExists
=== RUN   TestFileExists/Returns_true_when_given_file_exists
=== RUN   TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN   TestFileExists/Returns_false_when_provided_path_is_empty
=== RUN   TestFileExists/Returns_false_when_provided_path_is_not_a_file
--- PASS: TestFileExists (0.00s)
    --- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
    --- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_not_a_file (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils 0.349s
```

Notice we have no _testdata_ any more.
Basically, `Afero` allows you to not use `os` directly but write all your code in a way so that instead of the `os` package it calls methods on an instance of a type implementing particular interface: in the production code that object is a thin shim for the `os` package, and in the testing code it's replaced by whatever you wish; `afero` has a readily-available in-memory FS backend to do this.

To sum up, although it's a good idea to do this, there might be cases where Afero fails to offer you this isolation.
For example if you try to do anything on top of the basic filesystem usage, then Afero might not be enough.
For example, if you try to extract, zip or compress a fail, you will see that those functions are not compatible with Afero, thus you will have either to send a PR to afero or suffer by creating lot's of extra code for achieving that.