Learn from Durable Functions Architecture
Tags: azure, durablefunctions, queue, table
Published: 2021-08-19 18:56:20

[Durable Functions](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp) is a great serverless choice for **stateful** applications. And we can learn how to use queues and tables for loosely coupled stateful system if we want to develop our own system.

Durable Function depends on Durable Tasks framework which you can see source code in GitHub.

[Durable Functions Extension](https://github.com/Azure/azure-functions-durable-extension)
[Durable Task](https://github.com/Azure/durabletask)

I use Azure Storage as storage provider in this article.

# Use Queues to chain functions

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/orrp878o9w4frcqgoysz.png)

Durable Function uses normal Azure Function triggers to start **Client** function which triggers **Orchestration** functions and/or **Entity** functions. This is great design as we can use all supported trigger to integrate with Durable Functions.

Then we can call ``IDurableOrchestrationClient.Starter`` method to invoke Orchestrator function. The method actually doesn't call the function directly but create queue item to **control-queue**. 
 
See [TaskHubQueue.cs: AddMessageAsync](https://github.com/Azure/durabletask/blob/e8b169ac97d2a2a4926412a1dcffbbca5e4c0e78/src/DurableTask.AzureStorage/Messaging/TaskHubQueue.cs#L91) which calls ``storageQueue.AddMessageAsync`` to add queue item.

Same mechanism is used to trigger **Activity functions** and it uses another queue called **work item queue**.

# Use dispatcher to monitor queue

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6u7642xhfrjkrusei90r.png)

Durable Functions start TaskHubWorker at startup, which starts OrchestrationService. This service has mechanism to monitor queue.

[DurableTaskExtension.cs: StartTaskHubWorkerIfNotStartedAsync](https://github.com/Azure/azure-functions-durable-extension/blob/d8186d84c51a982929debf7b5b6186ca236d4822/src/WebJobs.Extensions.DurableTask/DurableTaskExtension.cs#L1198) call 
 [TaskHubWorker.cs: StartAsync](https://github.com/Azure/durabletask/blob/e8b169ac97d2a2a4926412a1dcffbbca5e4c0e78/src/DurableTask.Core/TaskHubWorker.cs#L148) which eventually call ``GetMessagesAsync`` to get queue item from queue.

See [ControlQueue.cs: GetMessagesAsync](https://github.com/Azure/durabletask/blob/e8b169ac97d2a2a4926412a1dcffbbca5e4c0e78/src/DurableTask.AzureStorage/Messaging/ControlQueue.cs#L48) for detail code.

Durable Task also has separate class to do the same for Activity Functions and Entity Functions.

### Cost and performance

As [queue-polling](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-perf-and-scale#queue-polling) section describes, polling queue incur costs, so you don't want to query often when there is no item in the queue. However, it you need to query often enough to meet performance requirement. Durable Function uses back-off algorithm to balance cost and performance by using ``UpdateDelay`` method.

See [BackoffPollingHelper.cs: UpdateDelay](https://github.com/Azure/durabletask/blob/e8b169ac97d2a2a4926412a1dcffbbca5e4c0e78/src/DurableTask.AzureStorage/BackoffPollingHelper.cs#L90) for detail algorithm.

# Use table to store history

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/84llod0shxd9fhkl46xb.png)
 
Durable Function uses Azure Storage table to track execution history which contains not only when each function has been planned/executed, but also input/output data.

[AzureTableTrackingStore.cs](https://github.com/Azure/durabletask/blob/main/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs) has related methods to insert/update data in storage.

For example, [AzureStorageOrchestrationService.cs: CreateTaskOrchestrationAsync](https://github.com/Azure/durabletask/blob/e8b169ac97d2a2a4926412a1dcffbbca5e4c0e78/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs#L1539) firstly get queue item from control-queue, invoke orchestrator function, then log history to table by using ``trackingStore``.

Durable Functions uses [**History table**](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestrations?tabs=csharp#history-table) and [**Instances Table**](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-perf-and-scale#instances-table) to persist information about execution history and latest instance information. 

Keep latest information separate from history is useful when you want to query current orchestration status, but you need to keep both table in sync.

Read [DurableTask.AzureStorage README](https://github.com/Azure/durabletask/tree/main/src/DurableTask.AzureStorage) to fully understand the details.

# Orchestration performance

There are several things you need to know about orchestrator function related to performance.

### Reading history table

Durable Functions uses **Orchestrator Function** as workflow which usually calls multiple **Activity Functions** to do actual work. As Azure Function is a **stateless** service, when orchestrator function invoked, it always restarts from first line of the method.

The orchestrator function reads its history information to determine which activities had executed and their output values so that it can continue from previous step. 

This means that orchestrator function needs to load entire history every time, which obviously may impact entire performance.

### Queue design

Durable Functions explicitly separate queues for orchestration and activities because these functions handles different types of workload. Orchestrator function manage workflow, which should be light-weighted, whereas Activity function runs actual task which could I/O and/or CPU intense codes. 

Durable Function uses multiple control queues for control-queues (for orchestrator function) and one work item queue. Number of control queues are defined as partition in configuration. See below diagram which has three partition as an example.

![](https://docs.microsoft.com/en-us/azure/azure-functions/durable/media/durable-functions-perf-and-scale/scale-interactions-diagram.png)

To fully utilize this architecture, you should keep orchestration workflow as simple as possible, and use **sub-orchestration** when you need to run large workflow, which is same concept as parent-child workflows.

Durable Function also uses [Task Hubs](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-task-hubs?tabs=csharp) as a logical container for functions and storages, but I won't explain it in this article.

# Did I cover everything?

Of course NOT. Durable Functions and Durable Task has way more features to support Enterprise scale application and system. I only explain some concepts but I believe it still provides some value. Visit each GitHub repository for more information or read official [docs](https://docs.microsoft.com/en-us/azure/azure-functions/durable/).

# Summary

What we can learn from Durable Functions Architecture? 

Durable Function architecture is great example of a **Cloud Native** application. It usually has multiple small components which are loosely coupled. We often use ServiceBus or Storage Queue to connect them, and use Cosmos DB, Storage Table or SQL Server to persist data. If you need to design complex system, I believe Durable Function architecture may give you some hints.