Improving the .NET API to work with the structure of the file system. Part 2. Manipulate filesystem objects.
Tags: dotnet, filesystem, tooling, io
Published: 2023-04-17 20:15:17

In the first part of the article, I described improved methods to enumerate file system objects. In this post, I will describe methods for manipulating them.

I note that there is a new possibility to enumerate file system objects. The `FileSystemTraversalOptions` enumerated type has three new flags: `ShowFileOccurrence`, `ShowDirectoryEnter`, `ShowDirectoryLeave` and their combinations. These flags affect the results of methods that enumerate file system objects of all types or partially for directories. The `ShowFileOccurrence` flag specifies the output of file objects. `ShowDirectoryEnter` specifies the output of a directory when entering it from its parent directory. The `ShowDirectoryLeave` flag sets the output of the directory when leaving it to the parent directory. It is easy to see that the combination of the `ShowFileOccurrence` and `ShowDirectoryEnter` flags specifies a *preorder* traversal (`PreorderTraversal`) of the file system object tree, while the combination of the `ShowFileOccurrence` and `ShowDirectoryLeave` flags specifies a *postorder* traversal (`PostorderTraversal`). It is also possible to combine the `ShowDirectoryEnter` and `ShowDirectoryLeave` (`ShowDirectoryOccurrence`) flags, which specifies the appearance of one directory twice: when entering it from the parent and when leaving it to the parent. Using methods with custom results, you can mark the output of file system elements with a `FileSystemTraversalMarker` enumerated value with values: `None`, `EnterDirectory`, `LeaveDirectory` using the `ToMarkedInfo` extension method on an object of type derived from `FileSystemInfo` resulting in an object of type `MarkedInfo<TInfo>`. Below is a code snippet to display the full path of files and subdirectories from all levels.

``` csharp

new DirectoryInfo(@"C:\Temp")
  .EnumerateFileSystemInfos(FileSystemTraversalOptions.ShowAllOccurrence,
    (fileSystemInfo, traversalMarker) => fileSystemInfo.ToMarkedInfo(traversalMarker))
  .ForEach(item => Console.WriteLine($"{item.Info.FullName} : {item.TraversalMarker}"));

```

Let's now move on to methods for manipulating a group of file system objects. There are both versions of methods with custom functionality specified by the calling method and methods that implement the following predefined operations: copying, moving, replacing and deleting files, moving, deleting, replicating (copying structure) and clearing (deleting empty) directories.
Each group can have up to 3 types of method implementations: regular synchronous, asynchronous (an asynchronous call to a synchronous method using a thread pool is implemented for each file system object) and parallel synchronous using PLINQ to parallelize work with files, but not with directories, which are performed sequentially. All input methods accept enumerations of both the file system objects of the FileSystemInfo, FileInfo, DirectoryInfo types and custom containers containing them.
Asynchronous methods (have the _Async_ suffix) return processing results through the `IAsyncEnumerable` interface. For synchronous methods, there are both implementations with deferred processing (they have the _Defer_ suffix) and direct processing of all elements. Elements in deferred processing methods are processed as they are retrieved by external code from the enumeration. Methods with direct processing return the result of processing all elements. As I said before, methods with parallel processing capability only process file objects in parallel. Such methods have an additional maxConcurrents parameter that specifies the maximum degree of parallelism. Below is a table of testing methods for element-by-element deleting and copying a directory with a 13 GB project containing about 200,000 files and 33,000 directories. The time of work with a local and network drive (WiFi connection) was measured. After each test, the pages that cache files were reset.

Element-by-element deletion.

| MaxConcurrents | Local drive | Network drive |
|---|----------|----------|
| - | 00:03:44 | 00:40:22 |
| 2 | 00:03:11 | 00:18:33 |
| 4 | 00:02:05 | 00:14:43 |
| 8 | 00:02:00 | 00:14:23 |

Element-by-element copying.

| MaxConcurrents | Local drive | Network drive |
|---|----------|----------|
| - | 00:11:22 | 01:20:59 |
| 2 | 00:09:29 | 00:41:47 |
| 4 | 00:07:46 | 00:26:09 |
| 8 | 00:07:21 | 00:19:51 |

Each method has a `FileSystemManipulationOptions` enumerated parameter that contains the following flags to control processing actions:
_EnsureDirectory_ - controls the creation of directories required for the operation. An operation's options type may implement the `IEnsuringOptions` interface, which must provide a list of required paths that will be created if none are present. This preprocessing is performed only for the directory marked with the value `FileSystemTraversalMarker.EnterDirectory`, i.e. before processing its child objects.
_CleanupDirectory_ - manages the removal of empty directories. This post-processing is performed only for the directory marked with the value `FileSystemTraversalMarker.LeaveDirectory`, i.e. after processing all of its child objects.
_ClearReadOnly_ - Controls the clearing of the ReadOnly attribute for file system objects that are about to be deleted or overwritten.
_Refresh_ - controls how the file system object's state is updated before it is used.
_SkipEmptyDirectory_ - skips any processing for empty directories.

Each method also contains an optional delegate for handling errors when working with file system objects - `errorHandler`. As input, it takes the original object, the operation with which caused an error, the parameters of the operation, and the exception that occurred. It can be used for logging. This delegate must return a boolean value that controls the suppression of the thrown exception and the continuation of processing of subsequent elements.

Let's look at calling methods of 6 operations (copy, move, replace, delete, replicate and cleanup) working on a directory containing files and subdirectories.

#### 1. Copy
The following code causes the entire structure (files and subdirectories) of the `srcDirectoryName` directory to be copied to the `dstDirectoryName` directory.
``` csharp
new DirectoryInfo(srcDirectoryName)
  .EnumerateFileSystemInfos(FileSystemTraversalOptions.PreorderTraversal)
  .CopyFileSystemInfos(FileSystemManipulationOptions.EnsureDirectory | FileSystemManipulationOptions.ClearReadOnly,
    fileSystemInfo => new TransferOptions
    {
      DestinationPath = Path.Combine(dstDirectoryName, PwrPath.GetRelativePath(srcDirectoryName, fileSystemInfo.FullName)),
      Overwrite = true,
      NoProcessing = fileSystemInfo is DirectoryInfo,
    });
```
Files are copied element by element. Non-existent directories in the destination will be created automatically according to the `EnsureDirectory` flag set. If the copied files exist at the destination, they will be overwritten according to the `Overwrite` option set in the `TransferOptions` operation's parameter structure. If existing files to be overwritten have the ReadOnly attribute set, then it will be cleared according to the `ClearReadOnly` flag set.

#### 2. Move
The following code causes the entire structure (files and subdirectories) of the `srcDirectoryName` directory to be moved to the `dstDirectoryName` directory.
``` csharp
new DirectoryInfo(srcDirectoryName)
  .EnumerateFileSystemInfos(FileSystemTraversalOptions.ShowAllOccurrence, (fileSystemInfo, traversalMarker) => fileSystemInfo.ToMarkedInfo(traversalMarker))
  .MoveFileSystemInfos(FileSystemManipulationOptions.EnsureDirectory | FileSystemManipulationOptions.CleanupDirectory | FileSystemManipulationOptions.ClearReadOnly,
    (fileSystemInfo, traversalMarker) => new TransferOptions
    {
      DestinationPath = Path.Combine(dstDirectoryName, PwrPath.GetRelativePath(srcDirectoryName, fileSystemInfo.FullName)),
      Overwrite = true,
      NoProcessing = fileSystemInfo is DirectoryInfo,
    });
```
Files are moved element by element. Non-existent directories in the destination will be created automatically according to the `EnsureDirectory` flag set. Empty directories in the original location will be removed according to the `CleanupDirectory` flag set. If files to be moved exist at the destination, they will be overwritten according to the `Overwrite` option set in the `TransferOptions` operation's parameter structure. If existing files to be overwritten or directories to be deleted have the ReadOnly attribute set, then it will be cleared according to the `ClearReadOnly` flag set.

#### 3. Replacement
The following code causes the files in the `dstDirectoryName` directory to be replaced with the corresponding files in the `srcDirectoryName` directory, and optionally to create a backup of the replaced files in the `bakDirectoryName` directory, creating the required subdirectory structure in it.
``` csharp
new DirectoryInfo(srcDirectoryName)
  .EnumerateFileSystemInfos(FileSystemTraversalOptions.ShowAllOccurrence, (fileSystemInfo, traversalMarker) => fileSystemInfo.ToMarkedInfo(traversalMarker))
  .ReplaceFileSystemInfos(FileSystemManipulationOptions.EnsureDirectory | FileSystemManipulationOptions.CleanupDirectory | FileSystemManipulationOptions.ClearReadOnly,
    (fileSystemInfo, traversalMarker) => new ReplaceOptions
    {
      DestinationPath = Path.Combine(dstDirectoryName, PwrPath.GetRelativePath(srcDirectoryName, fileSystemInfo.FullName)),
      DestinationBackupPath = Path.Combine(bakDirectoryName, PwrPath.GetRelativePath(srcDirectoryName, fileSystemInfo.FullName)),
      NoProcessing = fileSystemInfo is DirectoryInfo,
    });
```
Files are replaced and backed up element by element. Non-existent directories in the backup location will be created automatically according to the `EnsureDirectory` flag set. Empty directories in the original location will be removed according to the `CleanupDirectory` flag set. If existing files in the `srcDirectoryName` or `dstDirectoryName` directories or subdirectories of `srcDirectoryName` to be deleted have the ReadOnly attribute set, then it will be cleared according to the `ClearReadOnly` flag set.

#### 4. Deletion.
The following code calls for element-by-element deletion of objects in the `srcDirectoryName` directory.
``` csharp
new DirectoryInfo(srcDirectoryName)
  .EnumerateFileSystemInfos(FileSystemTraversalOptions.PostorderTraversal)
  .DeleteFileSystemInfos(FileSystemManipulationOptions.CleanupDirectory | FileSystemManipulationOptions.ClearReadOnly,
    fileSystemInfo => new DeleteOptions
    {
      NoProcessing = fileSystemInfo is DirectoryInfo,
    });
```
Files are deleted element by element. Empty directories in the original location will be removed according to the `CleanupDirectory` flag set. If existing files or directories to be deleted have the ReadOnly attribute set, then it will be cleared according to the `ClearReadOnly` flag set.


#### 5. Replication.
The following code causes a copy of the subdirectory structure from the `srcDirectoryName` directory to the `dstDirectoryName` directory to be created.
``` csharp
new DirectoryInfo(srcDirectoryName)
  .EnumerateDirectories(FileSystemTraversalOptions.PreorderTraversal)
  .ReplicateDirectories(FileSystemManipulationOptions.None,
    directoryInfo => new ReplicateOptions
    {
      DestinationPath = Path.Combine(dstDirectoryName, PwrPath.GetRelativePath(srcDirectoryName, directoryInfo.FullName)),
    });
```

#### 6. Cleanup.
The following code causes empty subdirectories of the `srcDirectoryName` directory to be deleted.
``` csharp
new DirectoryInfo(srcDirectoryName)
  .EnumerateDirectories(FileSystemTraversalOptions.PostorderTraversal)
  .CleanupDirectories(FileSystemManipulationOptions.None);
```

Extended examples of using methods with progress can be found in the [FileSystemExamples](https://github.com/vaseug/PowerLib.v2/blob/master/PowerLib.Examples/IO/FileSystemExamples.cs) class.

I note that the nuget package that contains the `FileSystemInfoExtension` class from namespace PowerLib.System.IO is called [VasEug.PowerLib.System](https://www.nuget.org/packages/VasEug.PowerLib.System) and has a MIT license.

P.S. If there is interest, I will probably plan to publish in the future a microservice that provides for each identity registered in it the allocation of its own virtual file space (empty or with a predefined structure) with the ability to upload and download files, copy, move, delete and share files and subdirectories, restrict rights and quotas.