Serverless, Durable functions, how YOU can learn to implement the Fan-out/fan-in pattern
Tags: serverless, tutorial, showdev, javascript
Published: 2019-06-18 22:31:58

---
title: Serverless, Durable functions, how YOU can learn to implement the Fan-out/fan-in pattern
published: true
description: 
tags: serverless, tutorial, showdev, javascript
cover_image: https://thepracticaldev.s3.amazonaws.com/i/wp5jjnfgdwiiaclr3vdx.jpg
---
Follow me on [Twitter](https://twitter.com/chris_noring), happy to take your suggestions on topics or improvements /Chris

> Durable functions enable you to write large complex application flows. The best part is that it uses the Serverless functions you know. You only need to concentrate on defining how things should flow.

In the first part [Part I - Durable functions](https://dev.to/azure/durable-functions-stateful-long-running-functions-in-serverless-part-i-5bm), we tried to learn different core concepts such as:

- **Orchestrator function**, this is the function containing your business flow
- **Activity function**, the function/s carrying out the actual work 
- **Client function**, the entry point 

We built an app in that first article that just executed a number of activity functions, one after the other. 
Executing functions that way has a name, *chaining* and is a known application pattern.

In this article we will look at: 

- **Application patterns**, let's take a closer look at the most common architecture patterns used with Durable Functions.
- **Fan-out/fan-in**, we will zoom in on the *Fan-out/fan-in* pattern in particular.
- **Lab**, as part of us learning the *Fan-out/fan-in* pattern we will build something with it to ensure we understand the core concepts

## Resources
- [Free account Azure account](https://azure.microsoft.com/en-gb/free/?wt.mc_id=devto-blog-chnoring) You will need to sign up on Azure to use Durable Functions
- [Creating your first durable function with JavaScript](https://docs.microsoft.com/en-us/azure/azure-functions/durable/quickstart-js-vscode?wt.mc_id=devto-blog-chnoring) Quickstart that takes you through creating a durable function
- [Durable functions concepts](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-concepts?wt.mc_id=devto-blog-chnoring) Read more here on concepts and patterns and how to implement said patterns.
- [Orchestrator function constraints](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-checkpointing-and-replay#orchestrator-code-constraints?wt.mc_id=devto-blog-chnoring) Constraints you need to be aware of.
- [Event handling with Durable functions](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-external-events?wt.mc_id=devto-blog-chnoring)
How to raise and handle events.
- [Application patterns](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-concepts?wt.mc_id=devto-blog-chnoring)
A description of the different application patterns you can implement


## Application patterns
There is more than one pattern we could be using with Durable Functions. Let's get a mile-high view of the most common application patterns we have at our disposal:

- **Chaining**, a sequence of functions executes in a specific order, this is the pattern we used in the first article of our series on Durable Functions
- **Fan-out/fan-in**, execute multiple functions in parallel and then wait for all functions to finish
- **Async HTTP APIs**, A common way to implement this pattern is by having an HTTP call trigger the long-running action. Then, redirect the client to a status endpoint that the client polls to learn when the operation is finished
- **Monitoring**, refers to a flexible, recurring process in a workflow. An example is polling until specific conditions are met
- **Human interaction**, Many automated processes involve some kind of human interaction. Involving humans in an automated process is tricky because people aren't as highly available and as responsive as cloud services. An automated process might allow for this by using timeouts and compensation logic

## Fan-out/fan-in
This pattern is best explained by the below picture:

![](https://thepracticaldev.s3.amazonaws.com/i/fdocp9b7ozis01gs73hl.png)

In this pattern, we start off executing the function `F1`. Thereafter we have three parallel functions that we want to execute before we can move on to function `F3`. The order in which we execute the three parallel functions doesn't matter. The point is that they all need to finish before we can move on.

There are many questions here like:

1. When would I ever use this pattern
2. If I use this pattern, how would I implement it?

### The *When*

Let's try to answer each question in turn. *When* would you use it? There are quite a lot of workflows in which this behavior manifests itself. A quite common domain might be an assembly line in a factory. Let's say we have a toy car factory. Imagine that you start by building the chassis, that would be step `F1`. Then it moves on to a station where 3 different assemblers each needs to add their own thing to this product. One person adds wheels, the second one doors and the third person adds an engine. Then when they are done it moves on to the last station F3 in which the toy car gets a coat of paint.

There you have it. Great looking car and a happy kid somewhere. :)

![](https://thepracticaldev.s3.amazonaws.com/i/00ev12e43qj3jy1v402k.jpg)

Now you might work somewhere where do you something less glamorous than making kids happy but the principle is the same. You have something consisting of several steps and you want some steps to be carried out in parallel and some sequentially. 

### The *How*

Now, let's try to answer the other question, the *how*. 

We got some nice constructs for this in Durable Functions, constructs that enable us to run things in parallel as well as enables us to wait for a group of activities until they are all done processing.

What are those constructs you ask? There are three of them:

1. Raise external event
2. Wait for external event
3. Decision logic

The first one here looks like this:

```js
await client.raiseEvent(instanceId, 'EventName', <value>);
```

The second construct looks like this:

```js
yield context.df.waitForExternalEvent("EventName");
```

The third construct looks like this:

```js
yield context.df.Task.all([gate, gate2, gate3]);
```

It needs some more explanation. It answers the question of how we should wait. The above says I will `gate`, `gate2` and `gate3` to all resolve before I take my next step. Used together it might look like this in the Orchestrator function:

```js
const gate = context.df.waitForExternalEvent("WheelsAddedEvent");

const gate2 = context.df.waitForExternalEvent("DoorsAddedEvent");

const gate3 = context.df.waitForExternalEvent("SteeringAddedEvent");

yield context.df.Task.all([gate, gate2, gate3]);

const result = yield context.df.callActivity("Send_Car_To_Be_Painted");
```

Now, the above says that any of the above events can happen in any order but we will only send our toy car to be painted if all needed Car components have been added. 

I can imagine what you are thinking right now. What about raising an event?. When do I do that? Well imagine at each assembly point that you do the actual work adding wheels, doors or steering and after you are done you call a REST endpoint that ends up raising its respective event. Let me show that in a picture:

![](https://thepracticaldev.s3.amazonaws.com/i/bqm3xqezsqyxzs7zbobl.png)

It might be still a bit blurry in understanding how to construct such a flow. Don't worry we will show that in our next headline.

## Lab - dinner meeting in the Shire

In this Lab, we will use all the concepts we just presented namely how to raise events, wait for events and introduce two different ways of doing decision logic. 

The theme for this demo is LOTR or Lord of the Rings and more specifically the start of the movie Hobbit. So what takes place there? Well it all starts with a dinner party in the Shire where all the members of a quest party meet up, has dinner together and then they venture off. Of course, they end up eating all of Bilbo's food supply but that's a different story. 

Now, the reason for choosing this scene to demonstrate Durable Functions is that it represents a *Fan-out/fan-in* pattern. Something initial takes place. Then one dinner guest after another arrives, the order doesn't matter. After they all have arrived and had dinner, they can finally proceed on their adventure.

> Ok, so we need to *orchestrate* this in Durable Functions so where do we start? 

Let's start by identifying what *events* we have. If you remember your Hobbit movie correctly the dinner guests arrive one by one. So a guest *arriving* is an event. We also have some decision logic in there. Once all the dinner guests have assembled they start talking about their big plan. So we need to wait for *all* to arrive before we can proceed. With that knowledge we can actually start hammering out our orchestrator logic, like so:

```js
const gate = context.df.waitForExternalEvent("BilboArrived");

const gate2 = context.df.waitForExternalEvent("DvalinArrived");

const gate3 = context.df.waitForExternalEvent("GandalfArrived");

yield context.df.Task.all([gate, gate2, gate3]);

const result = yield context.df.callActivity("Talk_Shop");
```

Above we have created three different events `BilboArrived`, `DvalinArrived`, `GandalfArrived` and lastly we have an activity we kick off `Talk_Shop` as soon as all dinner guests are in place.

> Yes yes, I know there were a ton more guests but the above will illustrate the principle :)

From what we learned from our overview image we can create normal Azure Functions with HTTP triggers that when done can raise events, so that's pretty much it, let's turn this into actual code next.

### Scaffold our project

We start out with invoking our command palette, either `CMD + SHIFT + P` or `View > Command Palette` and we choose the below

![](https://thepracticaldev.s3.amazonaws.com/i/8xjslxrhz4oc12ih0sra.png)


Next is to create a `HttpStart` function. We invoke the command palette yet again, opt to create an Azure Function and choose `Durable Functions HTTP Starter`.

After that, we choose to create an orchestrator function. We follow the same steps as above but we choose the one called `Durable Functions Orchestrator`. 

Then we choose to create a normal `Azure Function`, we choose it to be an HTTP trigger and we name it `QuestParty`.

Lastly, we create a `Durable Functions activity` and choose to name it `Talk_Shop`.  

Your directory should look something like this

![](https://thepracticaldev.s3.amazonaws.com/i/2ovodyi1otp5reigkiqf.png)

### Set up orchestrator

Ok, we already sketched what this one might look like, but here goes again:

```js
const taskGandalf = context.df.waitForExternalEvent("Gandalf");
const taskBilbo = context.df.waitForExternalEvent("Bilbo");
const taskDvalin = context.df.waitForExternalEvent("Dvalin");

yield context.df.Task.all([taskGandalf, taskBilbo, taskDvalin]);

const result = yield context.df.callActivity("Talk_Shop");

return result;
```

The above code says we are waiting for the events `Bilbo`, `Dvalin`, `Gandalf`, in no particular order and the following line says all three needs to have occurred before we can proceed:

```js
yield context.df.Task.all([taskGandalf, taskBilbo, taskDvalin]);
```

and yea our final act is to invoke the activity `Talk_Shop`:

```
const result = yield context.df.callActivity("Talk_Shop");
```

That's it for the orchestration.

### Set up HTTP triggered `QuestParty` function

Ok, so this function is triggered by HTTP. We can see that if we go into `QuestParty/function.json` and specifically this binding entry: 

```json
{
  "authLevel": "anonymous",
  "type": "httpTrigger",
  "direction": "in",
  "name": "req",
  "methods": [
    "get",
    "post"
  ]
}
```

BUT, we had to add one thing more to make this one play ball, namely this entry:

```json
{
  "name": "starter",
  "type": "orchestrationClient",
  "direction": "in"
}
```
This enables us to talk to an orchestration client instance and we will need that to raise an event. Now, lets look at the code next `QuestParty/index.js`:

```js
const df = require("durable-functions");

module.exports = async function (context, req) {
  context.log('Quest party member arrival');

  const { who, instanceId }  = req.query;

  const client = df.getClient(context);

  const fellowshipMembers = ['Gandalf', 'Bilbo', 'Dvalin'];
  const found = fellowshipMembers.find(m => who);

  if(!found)  {
      context.res = {
          status: 400,
          body: `Someone unknown called ${who} just entered Bilbos house, crap!`
      };
  } else {
      await client.raiseEvent(instanceId, who, true);
      context.res = {
          // status: 200, /* Defaults to 200 */
          body: `Another hungry member ${who} entered Bilbos house`
      };
  }
};
```

Now, there are two pieces of very important info that we grab from the query parameters, namely `who` and `instanceId`. `who` is us passing an argument like `Gandalf`, `Dvalin` or `Bilbo`. The `instanceId` is a reference to that specific function invocation instance. So if we want to affect a specific execution instance we need to know that specific id. Where does it come from though? The first time you call `HttpStart/index.js` we get an `instanceId`:

```js
module.exports = async function (context, req) {
    const client = df.getClient(context);
    const instanceId = await client.startNew(req.params.functionName, undefined, req.body);

    context.log(`Started orchestration with ID = '${instanceId}'.`);

    return client.createCheckStatusResponse(context.bindingData.req, instanceId);
};
```

Point being, if we want to invoke our REST API `QuestParty` we need to bring it two different parameters for it to call the correct execution instance but also pass the correct info back to the orchestration function. 

Enough theory at this point. Let's start this up and debug it.

### Debug

Ok, so the best way of understanding how something works is simply to debug it. We will do just that by hitting `Debug > Start Debugging`. 

This should give us the following:

![](https://thepracticaldev.s3.amazonaws.com/i/rbwnn3mihqpfw13lavhh.png)

We see above that we have two endpoints we can hit:

1. http://localhost:7071/api/orchestrators/{functionName}
This will hit our entry point and start the Orchestration
2. http://localhost:7071/api/QuestParty

Let's start with the first one and start our Orchestration by calling it like so:

```
http://http://localhost:7071/api/orchestrators/Orchestrator
```

![](https://thepracticaldev.s3.amazonaws.com/i/3yamqmmwofv7e29uax46.png)

We step through everything and receive the following in the browser:

![](https://thepracticaldev.s3.amazonaws.com/i/1pxa6xfgj50zbytpy8yi.png)

We've highlighted the important part, namely our execution identifier. If we want to refer to this specific function invocation we need to keep track of this. 

Remember how our orchestration have been told to wait for events `Gandalf`, `Bilbo` or `Dvalin`? It's time to trigger those events, doesn't matter which of the three we start with. Let's hit our other endpoint like this for example:

```
http://localhost:7071/api/QuestParty?instanceId={the id we saw in the browser}&who=Gandalf
```

Given the above example URL, we will trigger the event `Gandalf`, given how the code is written in `QuestParty/index.js`. So let's copy the `id` from the browser and hit the `QuestParty` URL in the browser and see what happens:

![](https://thepracticaldev.s3.amazonaws.com/i/5bn7fu56osk4btp7dmsf.png)

Next we should be hitting VS Code and our `QuestParty` code like so:

![](https://thepracticaldev.s3.amazonaws.com/i/8cwqmgp3qdo65cikh0qv.png)

We see that the next thing to happen is that our event `Gandalf` is about to be raised. So we let the debugger continue.

![](https://thepracticaldev.s3.amazonaws.com/i/xxtavhrd0dmh2sjj66dv.gif)

Let's make another browser call to `QuestParty` endpoint:

```
http://localhost:7071/api/QuestParty?instanceId={the id we saw in the browser}&who={Dvalin, Gandalf or Bilbo}
```

with `Dvalin` and `Bilbo` as args respectively for the parameter `who`. After we continue the debugger on each invocation we will end up in the orchestrator here:

![](https://thepracticaldev.s3.amazonaws.com/i/fv9kw9juqpeezdtwqcv8.png)

As you can see above our decision logic has been fulfilled, all three events `Gandalf`, `Bilbo` and `Dvalin` have all been raised which means:

```
yield context.df.Task.all(taskGandalf, taskBilbo, taskDvalin)
```
and that means we are no longer stopping at the above row, but we pass it and our last order of business is calling the activity `Talk_Shop`.

There we have it, everyone's here, ready to perform a quest.

![](https://thepracticaldev.s3.amazonaws.com/i/3615wjr57eff3lklnenx.gif)

### Decision logic

We are currently using the method `all()` on the `Task` class to determine when we can continue. *all* means that we have to wait for all the defined tasks to finish before we can continue. There is another useful method we could use instead namely `any()`. That simply means that if any of the above events occur we can continue. Imagine a situation where one of three managers needs to sign for an invoice then the method `any()` would be the way to go, like so:

```js
const taskSignedCEO = context.df.waitForExternalEvent("InvoiceSignedCEO");
const taskSignedCFO = context.df.waitForExternalEvent("InvoiceSignedCFO");
const taskSignedManager = context.df.waitForExternalEvent("InvoiceSignedManager");

yield context.df.Task.any([taskSignedCEO, taskSignedCFO, taskSignedManager]);

const result = yield context.df.callActivity("Set_Invoice_As_Processed");

return result;
```

## Summary

This time around we talked about Application patterns for Durable Functions. Something that in my opinion makes Durable functions really powerful and useful. We talked about the importance of knowing the `instanceId` or at least some kind of unique identifier to know what specific orchestration invocation you are working on, as long as it's something unique that allows you to come back and keep on working on the same order, assembly or whatever thing you are working on. 

Furthermore, we talked about the specific pattern *Fan-out/fan-in* and exemplified how that could be used in an assembly line as well as waiting for Dwarves/Hobbits/Sorcerers to arrive to a dinner party. Regardless of your type of business Durable functions can greatly help to orchestrate your business flows.

### Acknowledgements

Thank you [Anthony](https://dev.to/anthony) for your support in making this article happen :)