Tutorial: Create an Azure Blob Storage with SFTP integration
Tags: azure, sftp, serverless, tutorial
Published: 2022-03-30 01:16:03

## Introduction
Azure has announced a SFTP feature for Storage Accounts. Currently, the feature is still in preview and should therefore not be used for production workloads until it reaches GA.

The new Azure Storage Account SFTP feature provides a full managed PaaS service. Before that, an SFTP service had to be provided in other ways such as:
- **Custom implementation**: Development of a custom service that provides the SFTP functions, which then persists the files in a blob storage. For this use case Microsoft offers a template, consisting of an Azure Storage Account fileshare and a container instance. For more information, please checkout this [link](https://docs.microsoft.com/en-us/samples/azure-samples/sftp-creation-template/sftp-on-azure/).
- **Virtual Machine**: Deploy a VM with a SFTP service, either Windows or Linux. Especially for small use cases, this variant could be a overkill and depending on the configuration, this solution is also quite expensive. Furthermore, the administration overhead is significantly higher than with the integrated SFTP option.

Advantages of the PaaS variant: 

- :white_check_mark: Easy to setup & use
- :white_check_mark: Fully managed serverless service 
- :white_check_mark: Extremely cheap
 
I would like to emphasize the last point in particular. Azure Blob Storage is an extremely cost-effective storage option, here is an example calculation:

**Storage Account Configuration**:

|Configuration Property Name             |Value                  |
|----------------------------------------|-----------------------|
|Region                                  |West Europe            |
|Type                                    |Block Blob Storage     |
|Performance Tier                        |Standard               |
|Storage Account Type                    |General Purpose V2     |
|Access Tier                             |Hot                    |
|Redundancy                              |LRS                    |

**Storage Account Pricing**:

|Name            |Description                     |Price         |
|----------------|--------------------------------|--------------|
|Capacity        |100 GB                          |1.76 €        |
|Write Operations|10 x 10.000                     |0.49 €        |
|List and create Container Operation | 10x 10.000 |0.04 €        |
|Data Retrieval  |1000 GB                         |0.0 €         |
|Data Write      |1000 GB                         |0.0 €         |
|Support         |Included                        |0.0 €         |
 
Monthly cost: **2.78 €** 

As illustrated here, Azure Blob Storage is really affordable! In the example you get for not even 3€ a complete managed storage with many different features like SFTP support and many more!

> :warning: Disclaimer: This is an example configuration. Storage accounts are "pay as you go" and other prices may apply depending on configuration and region. For more precise estimation, I highly recommend using the [_Azure Pricing Calculator_](https://azure.microsoft.com/en-us/pricing/calculator/)

## Let's get started
### Prerequisites
- Azure Account with an active subscription. In case you do not have an Azure Account, go ahead and create one for free [here](https://azure.microsoft.com/en-us/free/)
- Azure CLI, you can either install the [Azure CLI locally](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) or use the [Azure Cloud Shell](https://docs.microsoft.com/en-us/azure/cloud-shell/overview).

> :information_source: If the Azure Cloud Shell is used, then the command az login is not needed. In the Cloud shell you are automatically logged in with the current user.
 
### Enable the SFTP preview feature
In order to use this new feature, it has to be activated in advance for the respective subscription:

![Register the SFTP support in Azure Blob Storage feature](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0fl367fxrvwgazgxaw9l.png)

1. Navigate to [Azure Portal](https://portal.azure.com/)
2. In the search bar, search for _Subscriptions_
3. Select your subscription (in my case, my subscription is named _Pay-As-You-Go_)
4. Inside your subscription, search for _Preview Features_
5. Inside the available features, search for "sftp" and register the _SFTP support in Azure Blob Storage_

**Azure CLI**:

Install the storage-preview extension:
```shell
az extension add -n storage-preview
```

Azure login:
```shell
az login
```

List available subscriptions and set active azure subscription:
```shell
az account list --all
az account set --subscription <subscription-id>
```

Enable SFTP feature:
```shell
az feature register --namespace Microsoft.Storage --name AllowSFTP 
```

Verify feature registration:
```shell
az feature show --namespace Microsoft.Storage --name AllowSFTP 
```

## Create and configure the Azure Storage Account
Microsoft provides a [demo template](https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.storage/storage-sftp) , in order to deploy the template click on the following button:

[![Deploy To Azure](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazure.svg?sanitize=true)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.storage%2Fstorage-sftp%2Fazuredeploy.json)

With this template we are ready to go and can test the SFTP feature. But of course we will also deploy everything from scratch:

**Create the Azure Storage Account:**
```shell
az storage account create -n <storage-account-name> \
-l <azure-location> \
-g <resource-group> \
--sku Standard_LRS \
--https-only=true \
--hierarchical-namespace=true \
--kind StorageV2
```

**Enable SFTP:**
```shell
az storage account update -g <resource-group> \
-n <storage-account> \
--enable-sftp=true
```

**Create an Azure Storage Container:**
```shell
az storage container create -n <container-name> \
--account-name <storage-account-name>
```

**Create local user for SFTP access:**
```shell
az storage account local-user create \
--account-name <storage-account> \
-g <resource-group> \
-n <local-user-name> \
--home-directory <home-dir> \
--permission-scope permissions=<permissions> service=blob resource-name=<container-name> \ 
--ssh-authorized-key key="<ssh-key>" \
--has-ssh-key true \
--has-ssh-password true
```
_Authentication:_
For authentication, there are two options available:
- Password
- SSH Key

In order to create or regenerate a password, use this command:
```shell
az storage account local-user regenerate-password \
--account-name <storage-account> \
-g <resource-group> \
-n <local-user-name>
``` 
There are many ways to generate SSH key pairs. Just use your preferred tool, alternatively the Azure CLI can be used:
```shell
az sshkey create --name "<ssh-key-name>" \
--resource-group "<resource-group>"
```
This command creates a new SSH key Azure resource:
![Azure SSH key resource](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ggxrbp3hp49o5osfdcpu.png)
 

 
For more details, checkout the [official documentation](https://docs.microsoft.com/en-us/azure/virtual-machines/ssh-keys-azure-cli#generate-new-keys).

> :warning: While you can enable both forms of authentication, SFTP clients can connect by using only one of them. Multifactor authentication, whereby both a valid password and a valid public and private key pair are required for successful authentication is not supported.

> :warning: You can't retrieve the generated password later, so make sure to copy the password, and then store it in a place where you can find it. If you lose this password, you'll have to generate a new one.

_Home directory:_
Home directory is only the initial directory that the connecting local user is placed in. Local users can navigate to any other path in the container they are connected to if they have the appropriate container permissions.
Format: _container/folder/..._

_Available container permissions:_
<table>
    <tr>
        <th>Permission</th>
        <th>Permission Code</th>
        <th>Description</th>
    </tr>
    <tr>
        <td>Read</td>
        <td>r</td>
        <td>
            <ul>
                <li>Read all file contents</li>
            </ul>
        </td>
    </tr>
    <tr>
        <td>Write</td>
        <td>w</td>
        <td>
            <ul>
                <li>Upload file</li>
                <li>Create directory</li>
                <li>Upload directories</li>
            </ul>
        </td>
    </tr>
    <tr>
        <td>List</td>
        <td>l</td>
        <td>
            <ul>
                <li>List contents within container</li>
                <li>List contents within directories</li>
            </ul>
        </td>
    </tr>
    <tr>
        <td>Delete</td>
        <td>d</td>
        <td>
            <li>Delete files/directories</li>
        </td>
    </tr>
    <tr>
        <td>Create</td>
        <td>c</td>
        <td>
            <ul>
                <li>Upload file if file doesn't exist</li>
                <li>Create directory if it doesn't exist</li>
                <li>Create directories</li>
            </ul>
        </td>
    </tr>
</table>

You can also specify multiple permission scopes: 
```shell
--permission-scope permissions=rw service=blob resource-name=container1 \
--permission-scope permissions=rwd service=file resource-name=share2
```
## Connect via SFTP to the Azure Storage Account
I will briefly go over the creation of the SFTP user again here. Lets get started! In my case, I created the following SFTP user:

```shell
az storage account local-user create --account-name <storage-account> \
-g <resource-group> \
-n mka \
--home-directory mystoragecontainer \
--permission-scope permissions=rwcdl service=blob resource-name=mystoragecontainer \
--has-ssh-password true
```

> :information_source: For the sake of simplicity, I only use password authentication. As mentioned above you can also choose SSH key authentication or even both (please note that you can't use both at the same time, you can authenticate either via password or SSH key).

In order to obtain the password of the newly created user, please use this command:
```shell
az storage account local-user regenerate-password \
--account-name <storage-account> \
-g <resource-group> \
-n mka 
```

> :warning: Do not forget to copy the password after creating it, it cannot be requested again. If you lose the password, you must create a new one.

To connect to the storage account we need the blob primary endpoint. This can be obtained with the following command:

```shell
az storage account show -n <storage-account> -g <resource-group>
```
The previous command gives the general info about the storage account. Search for the section "primaryEndoints" and copy the "blob" URL without the prefix https or http(for example <storage-account-name>.blob.core.windows.net):

```json
"primaryEndpoints": {
    "blob": "https://<storage-account-name>.blob.core.windows.net/",
    "dfs": "https://<storage-account-name>.dfs.core.windows.net/",
    "file": "https://<storage-account-name>.file.core.windows.net/",
    "internetEndpoints": null,
    "microsoftEndpoints": null,
    "queue": "https://<storage-account-name>.queue.core.windows.net/",
    "table": "https://<storage-account-name>.table.core.windows.net/",
    "web": "https://<storage-account-name>.z6.web.core.windows.net/"
  }
```
Now you can connect to the storage account with your ***preferred SFTP client**. In my case I use PowerShell with OpenSSH and password authentication:

```shell
sftp <storage-account>.mka@<storage-account>.blob.core.windows.net
```
> :information_source: The username of the SSH user is in following format: _storage-account.sftp-user-name_

> :information_source: You might be prompted to trust a host key. During the public preview, valid host keys are published [here](https://docs.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-host-keys).

After that you will be asked to authenticate yourself, just enter the password here.

After successfully authenticating, a file can be uploaded with this command:
```shell
put <path-to-file>
```
Verify the file upload either directly within the SFTP client or via the following Azure CLI command:

_Retrieve Azure Storage Account Keys (for authentication):_
```shell
az storage account keys list -n <storage-account> -g <resource-group>
```

The result of this should look something like this:
```json
[
  {
    "creationTime": "2022-03-27T17:51:51.501304+00:00",
    "keyName": "key1",
    "permissions": "FULL",
    "value": "<key-value-1>"
  },
  {
    "creationTime": "2022-03-27T17:51:51.501304+00:00",
    "keyName": "key2",
    "permissions": "FULL",
    "value": "<key-value-2>"
  }
]
```
Simply copy one of the keys.

_List files:_
```shell
az storage fs file list -f <container-name> \
--recursive true \
--account-name <storage-account> \
--account-key <insert-previous-copied-key-here>
```

The output should look something like this:
```json
[
  {
    "contentLength": 53759,
    "etag": "0x8DA11E2F03CABAB",
    "group": "$superuser",
    "isDirectory": false,
    "lastModified": "2022-03-30T20:19:21",
    "name": "<my-file>",
    "owner": "$superuser",
    "permissions": "rw-r-----"
  }
]
```
## Conclusion 
As mentioned before, Azure Storage Accounts are a very cost effective option to store files that now offer SFTP integration.
Since this variant is a so-called serverless, there are many exciting use-cases! Especially many legacy systems that work with files often offer an SFTP upload. These programs can now upload the files directly to the cloud and once the files are in the cloud they can be easily processed further, e.g. with an Azure function that processes the uploaded files with an "on-file-created" trigger.

If you made it this far, thanks for reading! Let me know your thoughts on this new feature.

### Resources
- [Storage Accounts Overview](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview)
- [SSH File Transfer Protocol (SFTP) support for Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-support)
- [Connect to Azure Blob Storage by using the SSH File Transfer Protocol (SFTP)](https://docs.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-support-how-to?tabs=azure-portal)
- [Host keys for SSH File Transfer Protocol (SFTP) support for Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-host-keys)
- [Limitations and known issues with SSH File Transfer Protocol (SFTP) support for Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-known-issues)