Image recognition with TensorFlow.js
Tags: javascript, webdev, machinelearning, tensorflow
Published: 2020-11-30 15:31:32

In this post I will show you how to create a simple image classifier, without any machine learning knowledge using a pretrained model form the TensorFlow team.

**Checkout the [demo](https://dog-breed.basilebong.com/) and the [source code](https://github.com/basilebong/demo-dog-breed)**.

## Table of contents

- [Table of contents](#table-of-contents)
- [What you need](#what-you-need)
- [Let's start!](#lets-start)
  - [Initializing the app](#initializing-the-app)
  - [File uploader](#file-uploader)
- [Image classification](#image-classification)
  - [Loading](#loading)
  - [Using the model](#using-the-model)
  - [Display the prediction](#display-the-prediction)
- [Congratulations](#congratulations)
- [Credits](#credits)

## What you need

1. Knowledge of JavaScript, CSS and HTML
2. A code editor (I recommend VS Code)
3. A local server (I recommend live server VS Code extension).

## Let's start!

### Initializing the app

Create a new folder and add 3 files:

```
.
├── app.css
├── app.js
└── index.html
```

Edit `index.html` and add the following code:

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>My web app</title>

    <!-- Custom style -->
    <link rel="stylesheet" href="app.css" />

    <!-- Google font -->
    <link rel="preconnect" href="https://fonts.gstatic.com" />
    <link
      href="https://fonts.googleapis.com/css2?family=Source+Sans+Pro&display=swap"
      rel="stylesheet"
    />
  </head>
  <body>
    <script src="app.js"></script>
  </body>
</html>
```

### File uploader

In order to classify an image we first need to let the user upload a picture.

Edit `index.html` and add the following code inside `<body></body>`:

```html
<main>
  <section class="image-section">
    <img src="" id="image" />
  </section>

  <section class="file-section">
    <div class="file-group">
      <label for="file-input">Upload a picture</label>
      <input type="file" id="file-input" />
    </div>
  </section>
</main>
```

Edit `app.css` to enhance the look:

```css
body {
  font-family: "Source Sans Pro", sans-serif;
}

main {
  width: 100%;
  min-height: 100vh;
  display: flex;
  justify-content: center;
  align-items: center;
  flex-direction: column;
}

section {
  margin: 2rem 1rem;
}

.file-group {
  padding: 1rem;
  background: #efefef;
  border-radius: 1rem;
}

#image {
  max-width: 100%;
  width: 400px;
  height: auto;
  display: block;
  margin: auto;
}

.image-section {
  display: none;
  position: relative;
}

.image-loaded .image-section {
  display: block;
}
```

The next step is to create the JavaScript code that will handle the file upload and display the image on the page.

To help us manipulate the image and the file input, we are going to save those two DOM elements into some variables.

```js
const fileInput = document.getElementById("file-input");
const image = document.getElementById("image");
```

When the user uploads a new image, the `getImage()` function is triggered.

```js
fileInput.addEventListener("change", getImageDataUrl);
```

The goal is to display the uploaded image inside our web application. To do so create a new function `getImage()` and write it **before the event listener**.

```js
function getImage() {
  // ...
}
```

First we have to check if the file has been uploaded with success. So add the following code inside the `getImage()` function.

```js
function getImage() {
  if (!fileInput.files[0]) throw new Error("Image not found");
  const file = fileInput.files[0];
}
```

Then we need to read the file that has been uploaded with `FileReader`. You can find more information on the [mozilla.org](https://developer.mozilla.org/fr/docs/Web/API/FileReader/readAsDataURL) webpage.

To display the image inside our web app, we need a URL that can be set as `src` attribute of the `<img id="image"/>` tag. This URL will be generated by the `readAsDataURL(file)` method that returns a data URL.

> Data URLs, URLs prefixed with the data: scheme, allow content creators to embed small files inline in documents. They were formerly known as "data URIs" until that name was retired by the WHATWG. -[mozilla.org](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs)

```js
const reader = new FileReader();
```

The `FileReader` is **asynchronous**. We have to wait for the result with `onload` before we can display the image.

```js
reader.onload = function (event) {
  image.setAttribute("src", event.target.result);
  document.body.classList.add("image-loaded");
};

reader.readAsDataURL(file);
```

Finally, your `app.js` file should look like this:

```js
const fileInput = document.getElementById("file-input");
const image = document.getElementById("image");

/**
 * Get the image from file input and display on page
 */
function getImage() {
  // Check if an image has been found in the input
  if (!fileInput.files[0]) throw new Error("Image not found");
  const file = fileInput.files[0];

  // Get the data url form the image
  const reader = new FileReader();

  // When reader is ready display image.
  reader.onload = function (event) {
    image.setAttribute("src", event.target.result);
    document.body.classList.add("image-loaded");
  };

  // Get data url
  reader.readAsDataURL(file);
}

/**
 * When user uploads a new image, display the new image on the webpage
 */
fileInput.addEventListener("change", getImage);
```

## Image classification

Thanks to TensorFlow and its pretrained model, the classification of images becomes very easy. A model is a file that has been trained over a set of data in order to recognize certain patterns. I will not deep dive into this subject, but if you want to know more I recommend you to read the [Microsoft documentation](https://docs.microsoft.com/en-us/windows/ai/windows-ml/what-is-a-machine-learning-model#:~:text=A%20machine%20learning%20model%20is,and%20learn%20from%20those%20data.).

To start using TenserFlow.js and it's pretrained image classification model (`mobilenet`) we will have to edit the `index.html` file and add the following lines into the `<head></head>`:

```html
<!-- TensorFlow-->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@1.0.1"></script>
<!-- TensorFlow pretrained model-->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/mobilenet@1.0.0"></script>
```

### Loading

To avoid that the web application is used before the model is fully loaded, we will display a loader inside our web application.

Edit `index.html`, add the class `.loading` to the `<body></body>`, and the HTML markup of the loader.

```html
<!-- Add loading class -->
<body class="loading">
  <main>
    <!-- Add this -->
    <div class="loader">
      <h2>Loading ...</h2>
    </div>

    <!-- ... -->
  </main>

  <script src="app.js"></script>
</body>
```

Next we will have to hide the file input during the loading process. To do so edit `app.css` and add the following code:

```css
.loading .loader {
  display: block;
}

.loader {
  display: none;
}

.loading .image-section,
.loading .file-section {
  display: none;
}
```

Next we will have to load the model in our `app.js` file. Add the following code at the end of your file.

```js
// Async loading
mobilenet.load().then(function (m) {
  // Save model
  model = m;

  // Remove loading class from body
  document.body.classList.remove("loading");

  // When user uploads a new image, display the new image on the webpage
  fileInput.addEventListener("change", getImage);
});
```

As you can see `addEventListener` has been moved inside the loading function. We also need to add an empty `model` variable at the beginning of our code:

```js
const fileInput = document.getElementById("file-input");
const image = document.getElementById("image");
let model;

// ...
```

Finally, your code should look like this:

```js
const fileInput = document.getElementById("file-input");
const image = document.getElementById("image");
let model;

/**
 * Get the image from file input and display on page
 */
function getImage() {
  // Check if an image has been found in the input
  if (!fileInput.files[0]) throw new Error("Image not found");
  const file = fileInput.files[0];

  // Get the data url form the image
  const reader = new FileReader();

  // When reader is ready display image
  reader.onload = function (event) {
    const dataUrl = event.target.result;
    image.setAttribute("src", dataUrl);
    document.body.classList.add("image-loaded");
  };

  // Get data URL
  reader.readAsDataURL(file);
}

/**
 * Load model
 */
mobilenet.load().then(function (m) {
  // Save model
  model = m;

  // Remove loading class from body
  document.body.classList.remove("loading");

  // When user uploads a new image, display the new image on the webpage
  fileInput.addEventListener("change", getImage);
});
```

Now the UI is only displayed when the model is fully loaded.

### Using the model

The `mobilenet` model needs an `<img />` HTML element as parameter that **has a defined width and height. Currently this two attributes are missing.** To add them we will have to edit the `getImage()` function inside the `app.js` file.

To get the size of the image we will use the `Image` class.

> The Image() constructor creates a new HTMLImageElement instance. It is functionally equivalent to document.createElement('img'). -[mozilla.org](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/Image)

```js
function getImage() {
  // ...
  reader.onload = function (event) {
    // ...

    // Create image object
    const imageElement = new Image();
    imageElement.src = dataUrl;

    // When image object is loaded
    imageElement.onload = function () {
      // Set <img /> attributes
      image.setAttribute("src", this.src);
      image.setAttribute("height", this.height);
      image.setAttribute("width", this.width);

      // Classify image
      classifyImage();
    };

    // ...
  };

  //..
}
```

The `classifyImage()` function **does not exist yet.**
Now your `getImage()` function should look like this:

```js
function getImage() {
  // Check if an image has been found in the input
  if (!fileInput.files[0]) throw new Error("Image not found");
  const file = fileInput.files[0];

  // Get the data url form the image
  const reader = new FileReader();

  // When reader is ready display image
  reader.onload = function (event) {
    // Ge the data url
    const dataUrl = event.target.result;

    // Create image object
    const imageElement = new Image();
    imageElement.src = dataUrl;

    // When image object is loaded
    imageElement.onload = function () {
      // Set <img /> attributes
      image.setAttribute("src", this.src);
      image.setAttribute("height", this.height);
      image.setAttribute("width", this.width);

      // Classify image
      classifyImage();
    };

    // Add the image-loaded class to the body
    document.body.classList.add("image-loaded");
  };

  // Get data URL
  reader.readAsDataURL(file);
}
```

After a lot of preparation we can finally use the model with only a view lines of code. First we will create a new function called `classifyImage()`.

```js
function classifyImage() {
  model.classify(image).then(function (predictions) {
    console.log("Predictions: ");
    console.log(predictions);
  });
}
```

Run the application and you should see the predictions in your developer console!

### Display the prediction

The last thing we want to do is to display a sentence that describes the picture.
First we need to add a place in our HTML code where the description can be placed.
Edit `index.html`:

```html
<!-- ... -->
<section class="image-section">
  <img src="" id="image" />
  <div class="image-prediction"></div>
</section>
<!-- ... -->
```

Then add the necessary CSS in `app.css`:

```css
/* Black overlay over the image */
.image-section::before {
  content: "";
  z-index: 2;
  position: absolute;
  height: 100%;
  width: 100%;
  background: linear-gradient(transparent, transparent, #000000);
}

.image-prediction {
  position: absolute;
  bottom: 1rem;
  text-align: center;
  font-size: 18px;
  color: #fff;
  left: 0;
  right: 0;
  z-index: 3;
}
```

Then open `app.js` and change the `classifyImage()` function:

```js
function classifyImage() {
  model.classify(image).then((predictions) => {
    displayDescription(predictions);
  });
}
```

The predictions are an array of predictions. Each prediction contains a `className` and a `probability`.

```js
[
  {
    className: "chow, chow chow",
    probabilty: 0.856542315,
  },
];
```

The first thing we are going to do is to sort the results and only keep the predictions with the height probability. In this case the probability needs to be at least 20% (which is super low). If it is lower we display an error message.

```js
function displayDescription(predictions) {
  const result = predictions.sort((a, b) => a > b)[0];

  if (result.probability > 0.2) {
    const probability = Math.round(result.probability * 100);

    // Display result
    description.innerText = `${probability}% shure this is a ${result.className.replace(
      ",",
      " or"
    )} 🐶`;
  } else description.innerText = "I am not shure what I should recognize 😢";
}
```

Finally, your code should look like this:

```js
const fileInput = document.getElementById("file-input");
const image = document.getElementById("image");
const description = document.getElementById("prediction");

let model;

/**
 * Display the result in the page
 */
function displayDescription(predictions) {
  // Sort by probability
  const result = predictions.sort((a, b) => a > b)[0];

  if (result.probability > 0.2) {
    const probability = Math.round(result.probability * 100);

    // Display result
    description.innerText = `${probability}% shure this is a ${result.className.replace(
      ",",
      " or"
    )} 🐶`;
  } else description.innerText = "I am not shure what I should recognize 😢";
}

/**
 * Classify with the image with the mobilenet model
 */
function classifyImage() {
  model.classify(image).then((predictions) => {
    displayDescription(predictions);
  });
}

/**
 * Get the image from file input and display on page
 */
function getImage() {
  // Check if an image has been found in the input
  if (!fileInput.files[0]) throw new Error("Image not found");
  const file = fileInput.files[0];

  // Get the data url form the image
  const reader = new FileReader();

  // When reader is ready display image
  reader.onload = function (event) {
    // Ge the data url
    const dataUrl = event.target.result;

    // Create image object
    const imageElement = new Image();
    imageElement.src = dataUrl;

    // When image object is loaded
    imageElement.onload = function () {
      // Set <img /> attributes
      image.setAttribute("src", this.src);
      image.setAttribute("height", this.height);
      image.setAttribute("width", this.width);

      // Classify image
      classifyImage();
    };

    // Add the image-loaded class to the body
    document.body.classList.add("image-loaded");
  };

  // Get data URL
  reader.readAsDataURL(file);
}

/**
 * Load model
 */
mobilenet.load().then((m) => {
  // Save model
  model = m;

  // Remove loading class from body
  document.body.classList.remove("loading");

  // When user uploads a new image, display the new image on the webpage
  fileInput.addEventListener("change", getImage);
});
```

## Congratulations

Congratulations, you did it!

Note that this application is not fully finished:

- We didn't check if the uploaded file is an image
- We didn't check if the image is a dog
- We didn't check for upload errors

## Credits

- [TensorFlow.js](https://www.tensorflow.org/js)
- [Data URL by mozilla.org](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs)
- [FileReader by mozilla.org](https://developer.mozilla.org/fr/docs/Web/API/FileReader/readAsDataUR)
- [Image by mozilla.org](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/Image)
- [What is a machine learning model? by Microsoft](https://docs.microsoft.com/en-us/windows/ai/windows-ml/what-is-a-machine-learning-model#:~:text=A%20machine%20learning%20model%20is,and%20learn%20from%20those%20data.)