Bring Machine Learning to the Browser With TensorFlow.js — Part III
Tags: tensorflow, javascript, machinelearning, webdev
Published: 2019-03-05 14:35:14

---
title: Bring Machine Learning to the Browser With TensorFlow.js — Part III
published: true
tags:  tensorflowjs,javascript,machinelearning,webdev 
canonical_url: https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3
---

> _cross-post from [medium.com](https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3)_

> _**Edited 2019 Mar 11** to include changes introduced in TensorFlow.js 1.0. Additional information about some of these TensorFlow.js 1.0 updates can be found [here](https://gist.github.com/caisq/3fc0beb6597f42d66be806c6692f310d)._

#### How to go from a web friendly format to a web application

![](https://cdn-images-1.medium.com/max/1024/1*quHPlo7HdaXmbx_h63ep7A.jpeg)<figcaption>“assorted-color textile leaves hanging decor” by <a href="https://unsplash.com/@designecologist?utm_source=medium&amp;utm_medium=referral">DESIGNECOLOGIST</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption>

Welcome to part three of a series of posts where I walk you through how TensorFlow.js makes it possible to bring machine learning to the browser. First, there’s an overview of how to [bring a pre-trained model into a browser application](https://dev.to/vabarbosa/bring-machine-learning-to-the-browser-with-tensorflowjspart-i-4f4m-temp-slug-2078470). Then you’ll find greater detail on how to [convert your pre-trained model to a web friendly format](https://dev.to/vabarbosa/bring-machine-learning-to-the-browser-with-tensorflowjspart-ii-1h6p-temp-slug-4911024). Now in this post, we step through using that web friendly model in a web application.

We continue with the [Image Segmenter](https://github.com/IBM/MAX-Image-Segmenter) from the [Model Asset Exchange (MAX)](https://developer.ibm.com/code/exchanges/models/) converted in [part two of this series](https://dev.to/vabarbosa/bring-machine-learning-to-the-browser-with-tensorflowjspart-ii-1h6p-temp-slug-4911024). The goal here is to give you a greater understanding of TensorFlow.js and how to utilize the model we made. We will create a basic web application without much style or additional libraries. To keep this article from getting too long and keep focus on TensorFlow.js, we will skip over the HTML and non-TensorFlow.js specific JavaScript code. But you can [review the complete application](https://github.com/vabarbosa/tfjs-sandbox/tree/master/image-segmenter/demo) on GitHub.

![](https://cdn-images-1.medium.com/max/890/1*67gIaFjypeft8OpBKCf0IA.png)<figcaption>image-segmenter application output</figcaption>

### Importing the model

The first step in importing the model to a browser readable format is to include the TensorFlow.js library in your HTML via script tag.

{% gist https://gist.github.com/vabarbosa/16b0b5faa32cb09fcf8029ceca3f9d3a %}

This will load the latest version of TensorFlow.js but you can also [target a specific version or load it via NPM](https://js.tensorflow.org/#getting-started).

With the library loaded, a global `tf` variable becomes available for accessing its API. You can load the Image Segmenter model using the [loadGraphModel](https://js.tensorflow.org/api/1.0.0/#loadGraphModel) API. 

> _**NOTE**: With TensorFlow.js version before 1.0 use the [loadFrozenModel](https://js.tensorflow.org/api/0.15.1/#loadFrozenModel) for API._

Pass the URL to the dataflow graph using the appropriate API.

**Using Tensorflow.js version 0.x.x**:
{% gist https://gist.github.com/vabarbosa/01deb82adbf5e48fe6655a2d43a10e64 %}

**Using TensorFlow.js version 1.x.x**:
{% gist https://gist.github.com/vabarbosa/e9a119d7f473adf76d119e4814731250 %}

Depending on the model size, loading may take some time. Once loaded, the model is ready to accept inputs and return a prediction.

### Pre-processing the input

Models will need the inputs to be of a specific type and/or size. In most cases, the input needs some sort of pre-processing before sending it to the model. For example, some models may require a one-dimensional array of a certain size while others may require more complex multi-dimensional inputs. So the input (e.g., image, sentence, etc.) would need to be pre-processed to the expected format.

For the Image Segmenter, recall, when inspecting the model graph, the input was an `ImageTensor`. It was of type and shape **uint8[1,?,?,3]**.

> This is a four-dimensional array of 8-bit unsigned integer values. The  **?** s are placeholders and can represent any length. They would correspond to the length and width of the image. The **1** corresponds to the batch size and the **3** corresponds to the length of the RGB value for a given pixel.

For 8-bit unsigned integer valid values are from 0 to 255. This corresponds with an image’s pixel RGB value which also ranges from 0 to 255. So, we should be able to take an image convert it to a multi-dimension array of RGB values and send that to the model.

To get a Tensor with the pixel values, you can use the [tf.browser.fromPixels](https://js.tensorflow.org/api/1.0.0/#browser.fromPixels) (or [tf.fromPixels](https://js.tensorflow.org/api/0.15.1/#fromPixels) for TensorFlow.js versions before 1.0) function provided by TensorFlow.js. This will produce three-dimensional array with the shape **[?, ?, 3]** from the given [HTMLImageElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement). But, the Image Segmenter is expecting a four-dimensional array. To insert an extra dimension and get the shape needed, you also will need to call the [expandDims](https://js.tensorflow.org/api/latest/#expandDims) function.

**Using Tensorflow.js version 0.x.x**:
{% gist https://gist.github.com/vabarbosa/e085304561ee05cdfe6a879e45c1c79d %}

**Using Tensorflow.js version 1.x.x**:
{% gist https://gist.github.com/vabarbosa/5a821e2242c4f67c215e7dd2d9e021d7 %}

You should now have the required input data to run the model.

### Running the model

Run the model by calling [predict](https://js.tensorflow.org/api/latest/#tf.Model.predict) with the input data. The function takes the input Tensor(s) and some optional configuration parameters. It returns a prediction.

{% gist https://gist.github.com/vabarbosa/3986ddc126ff46b7fafea720c9eedc7c %}

Computations are in batches. If needed you can run prediction on a single batch with the [predictOnBatch](https://js.tensorflow.org/api/latest/#tf.Model.predictOnBatch) function.

Depending on the model complexity, the prediction may take some time.

### Processing the output

The type and shape of the output returned depends on the model. To do something meaningful extra processing of the prediction is most likely required.

For the Image Segmenter, the output is a [segmentation map](https://github.com/IBM/MAX-Image-Segmenter#ibm-developer-model-asset-exchange-image-segmentation) with integers between 0 and 20. The integers corresponds to one of the pre-defined labels for each pixel in the input image.

In our web application, we are going to overlay the original image with the segments found. And each segment color coded. For example, RGB (192, 0, 0) for chairs and RGB (0, 64, 0) for potted plants.

To achieve this, start with the [dataSync](https://js.tensorflow.org/api/latest/#tf.Tensor.dataSync) Tensor function. The function will download the output tensor into a [TypedArray](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray). Next convert the TypedArray into a regular array with `Array.from`.

With a [color map](https://github.com/vabarbosa/tfjs-sandbox/blob/master/image-segmenter/assets/color-map.json), go through the converted array and assign the appropriate color to each segment. Then take this data to create the desired overlay image.

{% gist https://gist.github.com/vabarbosa/3a457546d09ff1b743c38fba67adb89a %}

You can now add the resulting image to your HTML page.

### Completing the web application

To complete the application, add buttons to load the model, upload an image, and run the model. Also, add the code to overlay the input image and output prediction.

![](https://cdn-images-1.medium.com/max/915/1*hS4mY8p_bmcv7s32jxg_fg.png)<figcaption>running image-segmenter application</figcaption>

You can find the completed project [here](https://github.com/vabarbosa/tfjs-sandbox/tree/master/image-segmenter). The repository contains the demo web application. It also includes the web friendly format from the [tensorflowjs\_converter](https://github.com/tensorflow/tfjs-converter). You will also find a Jupyter notebook to play with the Image Segmenter in Python.

### Machine Learning in JavaScript

The ability to use machine learning technology on the web is often limited. Creating and training some models involve massive data and intense computations. The browser may not be the ideal environment. But, an exciting use case is to take models trained elsewhere then import and run them in the browser.

With [TensorFlow.js](https://js.tensorflow.org) you can convert some of these models into a web friendly format. Then bring them into your web application. Making machine learning in JavaScript that much easier.

To check out even more interesting applications take a look at [Nick Kasten](https://medium.com/u/6c8f3f54d771)’s [magicat](https://github.com/CODAIT/magicat) or the TensorFlow.js version of his [Magic Cropping Tool](https://github.com/kastentx/MAX-Image-Segmenter-Web-App/tree/TFJS).

* * *