Working With Files and Folders in S3, Using AWS SDK for .NET
Tags: aws, s3, dotnet, files
Published: 2019-12-07 23:09:20

Introduction 
------------
 
In this tutorial, I am going to show you how to use AWS SDK for .NET to do some basic file operations on an S3 bucket. AWS provides both *low-level* and *high-level* APIs. First, we are going to see how to use low-level APIs and then we will perform the same operations using high-level APIs.
 
Prerequisite
------------


In order to run the following code, you need to install the [AWSSDK.S3][1] Nuget package.
 
I have created an S3 bucket named: `my-bucket-name-123` and I have created a folder named `my-folder` inside the bucket:

[![enter image description here][2]][2]


Low-Level APIs
--------------
 
The low-level APIs are mapped closely to the underlying REST API, here we use a Request object to provide request information and AWS responds with the Response object.
 

Understanding S3 Path (when using low-level API)
------------------------------------------------

First, we need to understand that there is no concept of folder in S3, everything is an object. If I want to create a folder called `sub-folder`, I need to append the folder name with a / to let AWS know that what I want is a folder, not a file... so I need to create:
 
   `my-s3-bucket-name-123/my-folder/sub-folder/`
 
If I don't include the trailing slash AWS will create an object called `sub-folder` instead of a folder.


Initializing `AmazonS3Client`
---------------------------
 
This is how we initialize S3 clients, which we are going to use for the remaining examples:

```
string bucketName = "my-bucket-name-123";  
string awsAccessKey = "AKI............";  
string awsSecretKey = "+8Bo..................................";  
  
IAmazonS3 client = new AmazonS3Client(_awsAccessKey, _awsSecretKey, 
RegionEndpoint.APSoutheast2);
```

*`AmazonS3Client` is [thread safe][3], you can make it static or use a singletone instance.*


Creating a folder
-----------------
 
Here we are going to create a folder called `sub-folder` inside `my-folder`:

```
string folderPath = "my-folder/sub-folder/";
  
PutObjectRequest request = new PutObjectRequest()  
{  
    BucketName = _bucketName,  
    Key = folderPath // <-- in S3 key represents a path  
};  
PutObjectResponse response = client.PutObject(request);  
```

*If you forget the trailing slash in the path (i.e. `"my-folder/sub-folder"`) it would create an object called `sub-folder`... if you include a slash at the beginning of the path (i.e. `"/my-folder/sub-folder/"`) it will create a folder with name as an empty string and put the remaining folders inside it.*


Copying a file into a folder
----------------------------
 
The following code would copy test.txt inside sub-folder:

```    
FileInfo file = new FileInfo(@"c:\test.txt");  
string path = "my-folder/sub-folder/test.txt";  

PutObjectRequest request = new PutObjectRequest()  
{  
    InputStream = file.OpenRead(),  
    BucketName = _bucketName,  
    Key = path // <-- in S3 key represents a path  
};  
PutObjectResponse response = client.PutObject(request);
```


Listing content of a folder
---------------------------
 
The following code would list the contents of `sub-folder`:

```
ListObjectsRequest request = new ListObjectsRequest  
{  
    BucketName = _bucketName,  
    Prefix = "my-folder/sub-folder/"  
};  
  
ListObjectsResponse response = client.ListObjects(request);    
foreach (S3Object obj in response.S3Objects)    
{    
    Console.WriteLine(obj.Key);    
}    
  
// result:  
// my-folder/sub-folder/  
// my-folder/sub-folder/test.txt  
```


Deleting file/folder
--------------------
 
In the following code first we delete `test.txt` and then `sub-folder`:

```
// delete test.txt file  
string filePath = "my-folder/sub-folder/test.txt";  

var deleteFileRequest = new DeleteObjectRequest  
{  
    BucketName = _bucketName,  
    Key = filePath  
};  
DeleteObjectResponse fileDeleteResponse = client.DeleteObject(deleteFileRequest);  
  
// delete sub-folder  
string folderPath = "my-folder/sub-folder/";  
var deleteFolderRequest = new DeleteObjectRequest  
{  
    BucketName = _bucketName,  
    Key = folderPath  
};  
DeleteObjectResponse folderDeleteResponse = client.DeleteObject(deleteFolderRequest);
```

High-level APIs
---------------
 
High-level APIs are designed to mimic the semantic of File I/O operations. They are very similar to working with `FileInfo` and `Directory`. 
 

Understanding the S3 path (when using high-level APIs)
------------------------------------------------------
 
When using high-level APIs, we need to use windows' styles paths, so use a backslash (NOT slash) in your path,
 
   `"my-folder\sub-folder\test.txt"`
 
Also note that, similar to low-level APIs we need a trailing backslash to indicate a folder, for example, `"my-folder\sub-folder\"` indicates that sub-folder is a folder whereas `"my-folder\sub-folder"` indicates that sub-folder is an object inside my-folder. 
 
Initializing `AmazonS3Client` 
-----------------------------
 
Use the same code as low-level APIs (above) to initialize `AmazonS3Client`.  


Creating a folder
-----------------
 
Here we are going to create a folder called `high-level-folder` and create another folder called `my-folder` inside it:

```
string path = @"high-level-folder";
 
S3DirectoryInfo di = new S3DirectoryInfo(client, _bucketName, path);  
if (!di.Exists)  
{  
    di.Create();  
    di.CreateSubdirectory("sub-folder");  
}  
```


Copying file into folder
------------------------
 
The following code would copy `test.txt` inside `sub-folder`:

```
FileInfo localFile = new FileInfo(@"c:\test.txt");  
string path = @"high-level-folder\sub-folder\test.txt";  
  
S3FileInfo s3File = new S3FileInfo(client, _bucketName, path);  
if (!s3File.Exists)  
{  
    using (var s3Stream = s3File.Create()) // <-- create file in S3  
    {  
        localFile.OpenRead().CopyTo(s3Stream); // <-- copy the content to S3  
    }  
}  
```


Listing content of a folder
---------------------------
 
The following code would list the content of a `sub-folder`:

```
string path = @"high-level-folder\sub-folder\";  
  
S3DirectoryInfo di = new S3DirectoryInfo(client, _bucketName, path);  
IS3FileSystemInfo[] files = di.GetFileSystemInfos();  
foreach (S3FileInfo file in files)  
{  
    Console.WriteLine($"{file.Name}");  
}  
  
// result:  
// test.txt  
```

*Unlike low-level API, here the folder name (sub-folder) is not listed.*


Deleting file/folder
--------------------
 
In the following code first we delete `test.txt` and then `sub-folder`:

```
// delete test.txt file
string filePath = @"high-level-folder\sub-folder\test.txt";

S3FileInfo s3File = new S3FileInfo(client, _bucketName, filePath);
if (s3File.Exists)
{
    s3File.Delete();
}
  
// delete sub-folder
string folderPath = @"high-level-folder\sub-folder\";

S3DirectoryInfo directory = new S3DirectoryInfo(client, _bucketName, folderPath);
if (directory.Exists)
{
    directory.Delete();
}
```


Wrapping up
-----------
 
When designing information systems, we follow a practice called [single source of truth][4] (SSOT), it ensures that every data element is edited in only one place. SSOT simplifies the information system and makes working with it a lot easier. Personally I like to extend this practice to every aspect of design... when designing a UI, there is no point in giving users 2 different ways to buy a product, it complicates the UI and confuses the user... in my opinion, the same is true for designing a software library. I think by providing 2 different APIs (low-level and high-level), AWS has unnecessarily complicated the process of communicating with the S3 bucket, especially because these APIs use different paths systems... personally I spent hours on an error because I was using a high-level style path, with low-level APIs.
 
Now, to make the matter worse, AWS is providing yet another way of interacting with S3 buckets, and that is using AWS TransferUtility which runs on top of low-level API and is the recommended way for reading/writing large objects (larger than a few GB). Have a look at [this AWS documentation][5] and see which one is the best option for your needs.

  [1]: https://www.nuget.org/packages/AWSSDK.S3/
  [2]: https://res.cloudinary.com/practicaldev/image/fetch/s--bQ4nQTHG--/c_imagga_scale,f_auto,fl_progressive,h_420,q_auto,w_1000/https://thepracticaldev.s3.amazonaws.com/i/1yz0xobui9py70gjl4m0.png
  [3]: https://stackoverflow.com/questions/7743488/is-the-amazon-net-aws-sdks-amazons3-thread-safe
  [4]: https://en.wikipedia.org/wiki/Single_source_of_truth
  [5]: https://aws.amazon.com/blogs/developer/the-three-different-apis-for-amazon-s3/