Dialog Flow, Google Home & an Internet Connected Swimming Pool - A Deep Dive Into Building a Pool Bot
Published: 2020-05-14 13:27:22


In this series we have covered [arduino hardware & sensors](https://dev.to/j4s0nc/hardware-sensors-a-deep-dive-into-building-a-pool-bot-51fh), [connecting a swimming pool to the internet](https://dev.to/j4s0nc/putting-my-swimming-pool-on-the-internet-a-deep-dive-into-building-a-pool-bot-7o1), [saving events to Azure Table Storage](https://dev.to/j4s0nc/saving-particle-events-to-azure-table-storage-a-deep-dive-into-building-a-pool-bot-2fn3) and [building a PWA using Vue](https://dev.to/j4s0nc/building-a-vue-pwa-a-deep-dive-into-building-a-pool-bot-g7h). 

Now let's talk about interfacing the pool with google assistant! 

## Dialog Flow

If you want to build Google Assistant integrations, you could use [Actions on Google](https://developers.google.com/assistant) directly, or you could check out [Dialog Flow](https://dialogflow.com/). 

Google has made it simple to write conversational experiences and handle the machine learning for you. We'll just define some intents and utterances. 

### Intents

What is the user trying to accomplish? What is their _intent_? For my case, I want to know the temperature in the pool. So I created an Intent called "Water Temperature". 

### Utterances

Next we need to defined a few training phrases. These are the things we think users will say to trigger our intent. 
  
![Dialog Flow Intent](https://dev-to-uploads.s3.amazonaws.com/i/6bc158h16uffbphogo0h.png)

### Responses 

When a user triggers this Intent by matching an utterance, we need to send a response. Dialog Flow allows us to hard code responses, but we want real time data from the pool bot! For that we'll enable Fulfilment. We can still define a hard coded list, this will be useful if our backend service is down or being slow. 

![Responses](https://dev-to-uploads.s3.amazonaws.com/i/6fgf789873fn1op1a35w.png)

### Fulfillment 

Switching over to the fulfillment tab, We'll add some webhook settings. You can dive into the [docs](https://cloud.google.com/dialogflow/docs/fulfillment-overview) for the full details, but basically Dialog Flow is going to POST a json object to us, and we will need to send back the response shape it's expecting. 

![Fulfillment](https://dev-to-uploads.s3.amazonaws.com/i/m446q2sv3ytomchfesq4.png)

## Azure Function Webhook

Earlier in this series [we talked about Azure Functions](https://dev.to/j4s0nc/saving-particle-events-to-azure-table-storage-a-deep-dive-into-building-a-pool-bot-2fn3#azure-functions), now we'll add one more endpoint to the backend services. 

Google offers Dialog Flow [client libraries](https://cloud.google.com/dialogflow/docs/reference/libraries) for several different languages. We'll grab the C# Nuget:

```
nuget install Google.Cloud.Dialogflow.V2
```

This gives us the request and response objects we'll need to deal with. Let's build a function that takes in the Dialog Flow `WebhookRequest`, a reference to the pool bot `CloudTable` data and we'll end up returning a Dialog Flow `WebhookResponse` object.

```csharp
using Google.Cloud.Dialogflow.V2;
using Microsoft.WindowsAzure.Storage.Table;
using PoolBot.Data.Storage;

public static class AiFulfillment
{
    public static async Task<WebhookResponse> HandleQuery(CloudTable dataTable, WebhookRequest request)
    {
        // Get latest data from cloud storage
        var sensorData = await TableStorageRepo.GetLatestSensorData(dataTable);

        // Build friendly messages to send back to Dialog Flow
        var textMsg = $"Pool temperature is {sensorData.IntakeTemp:#}°";
        var speech = $"The pool temperature is {sensorData.IntakeTemp:#}°";
        var solar = sensorData.ReturnTemp - sensorData.IntakeTemp;
        if(solar > 1)
        {
            speech += " and the solar panels are on";
        }
        var response = new WebhookResponse
        {
            FulfillmentText = speech,
            FulfillmentMessages =
            {                  
                new Intent.Types.Message
                {
                    SimpleResponses = new Intent.Types.Message.Types.SimpleResponses
                    {
                        SimpleResponses_ =
                        {
                            new Intent.Types.Message.Types.SimpleResponse
                            {
                                DisplayText = textMsg,
                                TextToSpeech = speech
                            }
                        }
                    }
                },
                new Intent.Types.Message
                {
                    BasicCard = new Intent.Types.Message.Types.BasicCard
                    {
                        Title =  $"{sensorData.IntakeTemp:#}°",
                        Subtitle =  solar > 1 ? $"+{solar:#.#}°" : "",
                        Image = new Intent.Types.Message.Types.Image
                        {
                            ImageUri = "https://poolbot.azurewebsites.net/img/water.gif"
                        }
                    }
                }
            }
        };

        return response;
    }


}
```

This function makes two messages and stuffs them in a `WebhookResponse` object. Why are we setting two objects into `FulfillmentMessages`? This allows us to do different things on different devices! If the user is talking to a Google Home speaker, they won't have a screen and will hear the `SimpleResponse` `TextToSpeech` read to them. If they have a Google Hub with a screen (or are using Assistant on their phone) Dialog Flow will use the `BasicCard` message. 

__Note About Google Object Serialization!__ For some dumb reason Google's object don't serialize correctly. If you just sent back their `WebhookResponse` object, you'd get an error from Dialog Flow. You need to `.ToString()` the response before sending it back, but keep in mind it _also_ needs to be a json object, not a string of json. So this may look hacky, but you have to ToString, then Deserialize the string back into a object (the correct object this time!). Then we can send the response:

```csharp
[FunctionName("AI_Fulfillment")]
public static async Task<HttpResponseMessage> Fulfillment(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = "AI/Fulfillment")] HttpRequestMessage req,
    [Table("Data", Connection = "AzureWebJobsStorage")] CloudTable dataTable,
    ILogger log)
{
    // Read the request into a Dialog Flow Request object
    var q = await req.Content.ReadAsAsync<WebhookRequest>();
    // Build a response to the incoming query
    var result = await AiFulfillment.HandleQuery(dataTable, q);
    // Use ToString to build the correct JSON needed for the response
    // Deserialize the JSON string back into an object and send it
    return req.CreateResponse(Newtonsoft.Json.JsonConvert.DeserializeObject(result.ToString()));
}
```

Here is [Demo Video](https://drive.google.com/file/d/1zUGBxMfiQiRL7K6Sxp6IVAzrTs-NJJtH) of this running on my Google Hub.