Helping Santa with the power of Azure AI
Tags: festivetechcalendar, cognitiveservices, ai, azure
Published: 2020-12-04 17:26:33

> This article is part of the [Festive Tech Calendar](http://festivetechcalendar.com/) initiative by Gregor Suttie and Richard Hooper. You'll find other helpful articles and tutorials published daily by community members and experts there, so make sure to check it out every day.

In this post, I take last year's [Xamarin Santa Talk Challenge](https://devblogs.microsoft.com/xamarin/santa-talk-xamarin-challenge/) as the base to improve a mobile application that relies on Azure Cognitive Services – Text Analytics and Azure Functions to determine if someone’s getting a gift.

![Santa Talk initial app](https://dev-to-uploads.s3.amazonaws.com/i/nzmw7vbqn4lyy8c5tsa2.gif)

As you can see, this application analyzes a letter (text) and determines its *sentiment*, i.e. a value between 0 and 1 that states if the content is positive (the score is closer to 1) or negative (score is closer to 0). An Azure Function connects the mobile app with the AI services to send the message and receive a response in return.

We start with the [initial code](https://github.com/codemillmatt/santa-talk) shared for the challenge and then perform the following changes in order to add another AI capability: Computer Vision to automatically read the letter (using OCR, Optical Character Recognition).

**Phase 1 - Creating an Azure resource **

**Step 1.** Open the [Azure Portal](https://portal.azure.com) and create a Computer Vision resource.

![Computer Vision Azure resource](https://dev-to-uploads.s3.amazonaws.com/i/xfcgm0pfrgpn8xycf117.png)

Alternatively, you can also create a Cognitive Services resource.

![Cognitive Services Azure resource](https://dev-to-uploads.s3.amazonaws.com/i/yxhmmfiq6wobe3viia4h.png)

**Step 2.** From the **Keys and Endpoint** section, copy the Key 1 and Endpoint.

![Obtaining the Cognitive Services key and endpoint](https://dev-to-uploads.s3.amazonaws.com/i/11cjmy7pz36b0qxozqha.png)

**Phase 2 - Modifying the Functions project**

**Step 1.** Open the **SantaTalk.Functions** project and add the Microsoft.Azure.CognitiveServices.Vision.ComputerVision Nuget package.

![Adding the Computer Vision Nuget package](https://dev-to-uploads.s3.amazonaws.com/i/l8so0nc0qpu5p2gzmwve.png)

**Step 2.** As indicated [here](https://github.com/codemillmatt/santa-talk#running-azure-functions-locally) with the Text Analytics service, In the local.settings.json file, add two variables: ComputerVisionAPIKey and ComputerVisionAPIEndpoint with the corresponding values obtained from Step 2 in the previous phase.

```
{
    "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "APIKey": "YOUR KEY WILL GO HERE",
    "APIEndpoint": "https://westus2.api.cognitive.microsoft.com/",
    "ComputerVisionAPIKey": "YOUR COMPUTER VISION KEY WILL GO HERE",
    "ComputerVisionAPIEndpoint": "https://aiinadaycognitive.cognitiveservices.azure.com/"
  }
}
```

**Step 3.** Add a new class to the project: ScanSanta, with the following code which provides a ScanSanta function that receives an image (the letter) and uses the **RecognizeTextInStreamAsync** method from the **ComputerVisionClient** class in order to detect the text from the letter. The auxiliary, private GetTextAsync method is used to poll for results by calling the GetTextOperationResultAsync method (also from ComputerVisionClient).

```
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using System.Text;

namespace SantaTalk.Functions
{
    public static class ScanSanta
    {
        static ComputerVisionClient visionClient;
        private const int numberOfCharsInOperationId = 36;

        static ScanSanta()
        {
            var keys = new ApiKeyServiceClientCredentials(Environment.GetEnvironmentVariable("ComputerVisionAPIKey"));

            visionClient = new ComputerVisionClient(keys) { Endpoint = Environment.GetEnvironmentVariable("ComputerVisionAPIEndpoint") };
        }

        [FunctionName("ScanSanta")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] Stream image,
            ILogger log)
        {
            var mode = TextRecognitionMode.Handwritten;
            var text = string.Empty;

            try
            {
                var result = await visionClient.RecognizeTextInStreamAsync(image, mode);
                text = await GetTextAsync(result.OperationLocation);
            }
            catch (Exception ex)
            {
                log.LogError(ex.ToString());

                return new StatusCodeResult(StatusCodes.Status500InternalServerError);
            }

            return new OkObjectResult(text);
        }

        private static async Task<string> GetTextAsync(string operationLocation)
        {
            var operationId = operationLocation.Substring(
                operationLocation.Length - numberOfCharsInOperationId);

            var result = await visionClient.GetTextOperationResultAsync(operationId);

            int i = 0;
            int maxRetries = 10;

            while ((result.Status == TextOperationStatusCodes.Running ||
                    result.Status == TextOperationStatusCodes.NotStarted) && i++ < maxRetries)
            {
                await Task.Delay(1000);
                result = await visionClient.GetTextOperationResultAsync(operationId);
            }

            var sb = new StringBuilder();

            foreach (var line in result.RecognitionResult.Lines)
            {
                foreach (var word in line.Words)
                {
                    sb.Append(word.Text);
                    sb.Append(" ");
                }

                sb.Append("\r\n");
            }

            return sb.ToString();
        }
    }
}
```

**Phase 3 - Modifying the mobile app**

**Step 1.** Add the **Xam.Plugin.Media** Nuget and **Xamarin.FFImageLoading.Transformations** packages in all the projects of the solution

![Nuget packages](https://dev-to-uploads.s3.amazonaws.com/i/a1ibrl7xragypuu6o4hw.png)

Now you have to configure the Android project. Everything that is mentioned from Steps 2-6 happens in the **SantaTalk.Android** project:

**Step 2.** Add a **MainApplication** class. The code goes as follows, and is used to interact with the current activity:

```
using System;

using Android.App;
using Android.Runtime;

using Plugin.CurrentActivity;

namespace SantaTalk.Droid
{
#if DEBUG
    [Application(Debuggable = true)]
#else
    [Application(Debuggable = false)]
#endif
    public class MainApplication : Application
    {
        public MainApplication(IntPtr handle, JniHandleOwnership transer)
            : base(handle, transer)
        {
        }

        public override void OnCreate()
        {
            base.OnCreate();
            CrossCurrentActivity.Current.Init(this);
        }
    }
}
```

**Step 3.** Add the following code to the **AndroidManifest.xml** inside the **<application>** tags:

```
<provider android:name="android.support.v4.content.FileProvider" 
          android:authorities="${applicationId}.fileprovider" 
          android:exported="false" 
          android:grantUriPermissions="true">
          
	  <meta-data android:name="android.support.FILE_PROVIDER_PATHS" 
                     android:resource="@xml/file_paths"></meta-data>
</provider>
```

**Step 4.** Add the following permissions to the same file (use the GUI): 

- Camera
- Read External Storage
- Write External Storage

**Step 5.** Create an **xml** folder under **Resources**, there, create a **file_paths.xml** file with the following content:

```
<?xml version="1.0" encoding="utf-8"?>
<paths xmlns:android="http://schemas.android.com/apk/res/android">
  <external-files-path name="my_images" path="Pictures" />
  <external-files-path name="my_movies" path="Movies" />
</paths>
```

**Step 6.** Add the **camera** and **photo** images to the drawable folders (under Resources). You can download them from [here](https://github.com/icebeam7/santa-talk/tree/master/src/SantaTalk.Android/Resources)

Now it's turn to set up the **SantaTalk.iOS** project (Steps 7 and 8)

**Step 7.** Add the following keys and strings to the **Info.plist** file:

```
  <key>NSCameraUsageDescription</key>
  <string>This app needs access to the camera to take photos.</string>
  <key>NSPhotoLibraryUsageDescription</key>
  <string>This app needs access to photos.</string>
  <key>NSMicrophoneUsageDescription</key>
  <string>This app needs access to microphone.</string>
  <key>NSPhotoLibraryAddUsageDescription</key>
  <string>This app needs access to the photo gallery.</string>
```

**Step 8.** Add the **camera and photo** images to the Resources folder. You can download them from [https://github.com/icebeam7/santa-talk/tree/master/src/SantaTalk.iOS/Resources](here). Include tthe 2x and 3x versions.

The final steps modify the **SantaTalk** project. 

**Step 10.** Inside the **Services** folder, create a **Base** folder that contains a new class **BaseService** that interacts with the two Functions (WriteSanta and ScanSanta). It is based on the **LetterDeliveryService** that already exists in the project.

```
using System;
using System.Net.Http;

using Xamarin.Essentials;

namespace SantaTalk.Services.Base
{
    public class BaseService
    {
        //string santaUrl = "{REPLACE WITH YOUR FUNCTION URL}/api/";

        protected string santaUrl = "http://localhost:7071/api/";
        protected static HttpClient httpClient = new HttpClient();

        public BaseService()
        {
            // if we're on the Android emulator, running functions locally, need to swap out the function url
            if (santaUrl.Contains("localhost") && DeviceInfo.DeviceType == DeviceType.Virtual && DeviceInfo.Platform == DevicePlatform.Android)
            {
                santaUrl = santaUrl.Replace("localhost", "10.5.132.243");
            }

            httpClient.BaseAddress = new Uri(santaUrl);
        }
    }
}
```

**Step 11.** Now, actually, modify the LetterDeliveryService. A lot of its code has already been implemented in the BaseService, so it is easier to simply replace it with the following code. It does exactly the same it was doing before (call the WriteSanta Function), but now inheriting the functionality from **BaseService**:

```
using System;
using System.Net.Http;
using System.Threading.Tasks;

using Newtonsoft.Json;

using SantaTalk.Models;
using SantaTalk.Services.Base;

namespace SantaTalk
{
    public class LetterDeliveryService : BaseService
    {
        public async Task<SantaResults> WriteLetterToSanta(SantaLetter letter)
        {
            SantaResults results = null;
            try
            {
                var letterJson = JsonConvert.SerializeObject(letter);

                var httpResponse = await httpClient.PostAsync("WriteSanta", new StringContent(letterJson));

                results = JsonConvert.DeserializeObject<SantaResults>(await httpResponse.Content.ReadAsStringAsync());
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine(ex);
            }

            if (results == null)
                results = new SantaResults { SentimentScore = -1 };

            return results;
        }
    }
}
```

**Step 12.** In the Services folder, create a **LetterScanService** class, that is similar to the previous one and will call the ScanSanta Function service

```
using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;

using SantaTalk.Services.Base;

namespace SantaTalk
{
    public class LetterScanService : BaseService
    {
        public async Task<string> ScanLetterForSanta(Stream image)
        {
            string results = string.Empty;
            try
            {
                var httpResponse = await httpClient.PostAsync("ScanSanta", new StreamContent(image));

                results = await httpResponse.Content.ReadAsStringAsync();
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine(ex);
            }

            if (string.IsNullOrWhiteSpace(results))
                results = "The letter is illegible";

            return results;
        }
    }
}
```

**Step 13.** In the same folder, create a **PhotoService** class that uses the MediaPlugin package that was installed from the very beginning in order to allow the user to either select a photo from the device or take a new one with the camera:

```
using System;
using System.Threading.Tasks;

using Plugin.Media;
using Plugin.Media.Abstractions;

using Plugin.Permissions;
using Plugin.Permissions.Abstractions;

namespace SantaTalk
{
    public class PhotoService
    {
        private PermissionStatus cameraOK;
        private PermissionStatus storageOK;

        private async Task Init()
        {
            await CrossMedia.Current.Initialize();

            cameraOK = await CrossPermissions.Current.CheckPermissionStatusAsync(Permission.Camera);
            storageOK = await CrossPermissions.Current.CheckPermissionStatusAsync(Permission.Storage);

            if (cameraOK != PermissionStatus.Granted || storageOK != PermissionStatus.Granted)
            {
                var status = await CrossPermissions.Current.RequestPermissionsAsync(new[] { Permission.Camera, Permission.Storage });
                cameraOK = status[Permission.Camera];
                storageOK = status[Permission.Storage];
            }
        }

        public async Task<MediaFile> TakePhoto()
        {
            await Init();

            MediaFile file = null;

            if (cameraOK == PermissionStatus.Granted
                && storageOK == PermissionStatus.Granted
                && CrossMedia.Current.IsCameraAvailable
                && CrossMedia.Current.IsTakePhotoSupported)
            {
                var options = new StoreCameraMediaOptions()
                {
                    Directory = "SantaTalk",
                    Name = $"{Guid.NewGuid()}.jpg",
                    SaveToAlbum = true
                };

                file = await CrossMedia.Current.TakePhotoAsync(options);
            }

            return file;
        }

        public async Task<MediaFile> ChoosePhoto()
        {
            MediaFile file = null;

            if (CrossMedia.Current.IsPickPhotoSupported)
                file = await CrossMedia.Current.PickPhotoAsync();

            return file;
        }
    }
}
```

**Step 14.** Open the **MainPageViewModel** class (under ViewModels folder) and...

A. Add the System.Threading.Tasks namespace:
```
using System.Threading.Tasks;
```

B. Create an ScanLetterCommand ICommand:
```
public ICommand ScanLetterCommand { get; }
```

C. Create the ScanLetterForSanta method:
```
private async Task ScanLetterForSanta(bool useCamera)
{
    var photoService = new PhotoService();

    var photo = useCamera ? await photoService.TakePhoto() : await photoService.ChoosePhoto();

    var scanService = new LetterScanService();
    var scannedLetter = await scanService.ScanLetterForSanta(photo.GetStream());

    LetterText = scannedLetter;
}
```

D. In the constructor, create an instance of ScanLetterCommand:
```
ScanLetterCommand = new Command<bool>(async (useCamera) =>
{
    await ScanLetterForSanta(useCamera);
});
```

**Step 15.** Finally, in the **MainPage.xaml** file, add the following code:

A. The FFImageLoading.Forms namespace (top section):

```
    xmlns:ffimageloading="clr-namespace:FFImageLoading.Forms;assembly=FFImageLoading.Forms"
```

B. A couple of boolean resources (below the BindingContext):
```
    <ContentPage.Resources>
        <x:Boolean x:Key="FalseValue">False</x:Boolean>
        <x:Boolean x:Key="TrueValue">True</x:Boolean>
    </ContentPage.Resources>
```

C. Replace the Label control that contains the text "WRITE YOUR LETTER TO SANTA" with the following controls that allows the user to use the camera or select an image from the device to provide a picture of the letter:
```
<StackLayout Orientation="Horizontal" Spacing="8">
    <Label Text="WRITE YOUR LETTER TO SANTA" FontSize="12" VerticalOptions="Center"/>

    <ffimageloading:CachedImage Source="camera.png" Aspect="AspectFill" DownsampleToViewSize="True">
        <ffimageloading:CachedImage.GestureRecognizers>
            <TapGestureRecognizer Command="{Binding ScanLetterCommand}" CommandParameter="{StaticResource TrueValue}" />
        </ffimageloading:CachedImage.GestureRecognizers>
    </ffimageloading:CachedImage>
                                
    <ffimageloading:CachedImage Source="photo.png" Aspect="AspectFill" DownsampleToViewSize="True">
        <ffimageloading:CachedImage.GestureRecognizers>
            <TapGestureRecognizer Command="{Binding ScanLetterCommand}" CommandParameter="{StaticResource FalseValue}" />
        </ffimageloading:CachedImage.GestureRecognizers>
    </ffimageloading:CachedImage>
</StackLayout>
```

Compile the app and execute it (the Functions must also be running locally, or you can publish them to Azure). If everything goes as planned, this is the expected output:

![Reading a letter using Computer Vision](https://dev-to-uploads.s3.amazonaws.com/i/pjvxq9n1gui47aviudzl.gif)

Full code is available on my [GitHub](https://github.com/icebeam7/santa-talk/) repository

As you can see, you can now select a photo of a letter and the app will scan it using AI! The text will be entered, and by clicking on the button, the sentiment will be calculated!

Azure Cognitive Services is amazing! If you want to learn more, [this is a good starting point](https://docs.microsoft.com/en-us/learn/paths/evaluate-text-with-language-services/).

I hope that this entry was interesting and useful for you. I invite you to visit [my blog](https://www.luisbeltran.mx/) for more technical posts about _Xamarin, Azure, and the .NET ecosystem_. I write in Spanish language =)

Thanks for your time, and enjoy the rest of the [Festive Tech Calendar](http://festivetechcalendar.com/) publications!

See you next time,
Luis