How to use Netherite Storage Provider in Durable Functions
Tags: azure, functions, severless, workflow
Published: 2021-07-20 16:23:47

Netherite Storage Provider is a backend implementation of Durable Functions that delivers high performance.

It's still a preview version, but if you are using Functions Premium (EP1-3), it's worth evaluating it now.

{% github microsoft/durabletask-netherite %}

- [Durable Functions storage providers - Azure | Microsoft Docs](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-storage-providers#comparing-storage-providers)

The use of Netherite requires some additional work, so we will briefly introduce how to use it.

## Install Netherite

First, install the `Microsoft.Azure.DurableTask.Netherite.AzureFunctions` package in your Azure Functions project.

- [NuGet Gallery | Microsoft.Azure.DurableTask.Netherite.AzureFunctions](https://www.nuget.org/packages/Microsoft.Azure.DurableTask.Netherite.AzureFunctions/)

```posh
Install-Package Microsoft.Azure.DurableTask.Netherite.AzureFunctions -Version 0.4.0-alpha
```

```sh
dotnet add package Microsoft.Azure.DurableTask.Netherite.AzureFunctions --version 0.4.0-alpha
```

After installing the package, declare that you want to use Netherite in `host.json`. There are many configurations in the `storageProvider`, please refer to the official documentation.

```json
{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": false,
        "excludedTypes": "Request"
      }
    },
  },
  "extensions": {
    "durableTask": {
      "hubName": "<YourHubName>",
      "useGracefulShutdown": true,
      "storageProvider": {
        "type": "Netherite",
        "StorageConnectionName": "AzureWebJobsStorage",
        "EventHubsConnectionName": "EventHubsConnection"
      }
    }
  }
}
```

Set the connection string names for Storage and Event Hubs to the key names in the Azure Functions App Settings or `local.settings.json`.

## Preserve `System.Threading.Channels.dll`

The current version `0.4.0-alpha` does not work with an error when deployed to Azure Functions, so you need to add `FunctionsPreservedDependencies` to the csproj file to avoid this problem.

```xml
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.DurableTask.Netherite.AzureFunctions" Version="0.4.0-alpha" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.5.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.13" />
  </ItemGroup>
  <ItemGroup>
    <FunctionsPreservedDependencies Include="System.Threading.Channels.dll" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>
```

Add a setting to prevent `System.Threading.Channels.dll` from being removed on deployment. This fix will allow it to work on Azure Functions.

This problem is expected to be fixed in a future version.

## Create Function App and Event Hubs

Create Function App and Event Hubs Namespace by using Azure Portal or ARM Template.

- Functions Premium (EP1-3)
  - Consumption Plan is not supported.
- Event Hubs Namespace (Standard, 1TU)
  - Event Hubs are automatically created by Netherite

Since Netherite creates the Event Hub automatically, it is easier to use `RootManageSharedAccessKey` for the SAS key.

## Use 64-bit worker process

FASTER needs to run in a 64-bit process, so change the Platform setting of Azure Functions to 64-bit.

![64-bit worker process](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bqgm7x7voq1tls9ziue4.png) 

If you run the application with the 32-bit Platform setting, the following exception will be thrown at runtime and it will not work properly.

```
Part** !!! Encountered exception while working on store in StoreWorker.Process: FASTER.core.FasterException: Overflow in AddressInfo - consider running the program in x64 mode for larger address space support
```

Note that when you deploy Azure Functions (Windows) from Azure Portal, 32-bit Platform will be selected by default.

## Enable Runtime Scale Monitoring

Durable Functions with Netherite installed running on Functions Premium (EP1-3) need to have Runtime Scale Monitoring enabled in order to scale out.

![Runtime Scale Monitoring](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/onbhrzz57vecjdp6raxe.png)

Since Runtime Scale Monitoring requires warmed-up instances, set `preWarmedInstanceCount` to `1` when creating resources with ARM Template or Terraform.

After these steps, Azure Functions using Netherite is now working perfectly. Now you can verify the performance and have fun.