😎 Control the expressions 😀 of your emoji avatar 😳
Tags: javascript, machinelearning, tutorial
Published: 2019-12-31 16:15:22

Face-api.js is an amazing library, it is a JavaScript face recognition API for the browser and nodejs implemented on top of *tensorflow.js* core.
With *face-api.js* you can:
- detect faces from an image;
- retrieve key landmarks (68) on the face and tracking them (mouth, nose, eyes etc);
- detect faces characteristics (each faces is decoded via 128 numbers) useful for find similarity between two faces;
- detect expression: neutral, happy, sad, angry, fearful, disgusted, surprised.

The GitHub repository of face-api.js is: [https://github.com/justadudewhohacks/face-api.js](https://github.com/justadudewhohacks/face-api.js)

The most important thing is that someone already performed the training on  a meaningful sample for the tasks explained above, so you can reuse the "models".

## The PoC: 😎🙂😀😥😠😨🤢😳
I would like to walk through a small Web App that allow you to control your avatar with your face expression.
The steps are:
- init the Project
- acquire the video stream through the Web Cam via getUserMedia;
- load models for face recognition and expression recognition;
- detect all Faces in the video streaming;
- retrieve the expression for each faces;
- display an emoji based on the most rated expression.


## Init the project
As first step you need to create a new *package.json* file in a new empty directory. Fill package.json with:
```json
{
  "name": "face-poc",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "face-api.js": "^0.22.0"
  }
}
```
The relevant part is the dependecy with *face-api.js*.

Then, you can perform a classic npm install

```sh
npm i
```

## Training / Models
You need to retrieve the models used by face-api.js.
You can download models from this URL: https://github.com/justadudewhohacks/face-api.js/tree/master/weights and store them in *models* directory.

## The layout

The HTML file is very simple.
It contains a video tag for display the WebCam video streaming, and a div tag where to place the emoji.
So create a *index.html* file

```html
<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>Emoji Expression Controller</title>
  <!-- 001 - include a style.css -->
  <link rel="stylesheet" type="text/css" href="style.css">
  <!-- 002 - include face-api.min.js -->
  <script defer src="./node_modules/face-api.js/dist/face-api.min.js"></script>
  <!-- 003 - include our script -->
  <script defer src="./script.js"></script>
</head>

<body>
  <div class="header">Emoji Expression Controller</div>
  <div class="content">
    <!-- 004 - define a tag for video -->
    <video id="cam" width="512" height="256" autoplay muted></video>
    <!-- 005 - define a div where to place the emoji -->
    <div id="face"></div>
  </div>
</body>

</html>
```

The relevant parts are:
- 001 include an external CSS where we will place the styles;
- 002 include (defer) the face-api installed in node-moduels (minificated version);
- 003 include our javascript file, where we will place our JS code;
- 004 define thee tag video with the *id*, *width*, *height*, *autoplay* and *muted* attributes;
- 005 define a div with the *id* attribute.


## A touch of style

Create a *style.css* file and fill it:

```css
body {
  margin: 0px;
  padding: 0px;
  width: 100vw;
  height: 100vh;  
  justify-content: center;
  align-items: center;
}
.header {
  font-size:  48px;
  font-weight: 800;
  justify-content: center;
  align-items: center;
  display: flex;
}
.content {
  justify-content: center;
  align-items: center;
  display: flex;
}
#face {
  font-size: 128px;
}

```

## The Logic

Create a *script.js* file where we will:
- load the needed models via *Promise*;
- access to webcam through *getUserMedia*;
- detect faces with expression every 500 milliseconds;
- map the expression with the right emoji;
- display the emoji.

```javascript
// 001 - Access to DOM for video and face icon
const video = document.getElementById('cam');
const face = document.getElementById('face');

// 002 - Load models for Face Detection and Face Expression
Promise.all(
  [
    faceapi.nets.tinyFaceDetector.loadFromUri('/models'),
    faceapi.nets.faceExpressionNet.loadFromUri('/models')
  ]
).then(startvideo)


async function startvideo() {
  console.info("Models loaded, now I will access to WebCam")
  // 003 - Access to Cam and display it on video DIV
  const stream = await navigator.mediaDevices.getUserMedia({
    video: true
  })
  video.srcObject = stream

}

// 004 - Define the array with emoji
let statusIcons = {
  default: '😎',
  neutral: '🙂',
  happy: '😀',
  sad: '😥',
  angry: '😠',
  fearful: '😨',
  disgusted: '🤢',
  surprised: '😳'
}

function detectExpression() {
  // 005 - Set the default Emoji
  face.innerHTML = statusIcons.default
  // 006 - setInterval to detect face/espression periodically (every 500 milliseconds)
  const milliseconds = 500
  setInterval(async () => {
    // 007 - Wait to detect face with Expression
    const detection = await
      faceapi.detectAllFaces(video, new faceapi.TinyFaceDetectorOptions())
        .withFaceExpressions()
    // 008 - detectAllFaces retruns an array of faces with some interesting attributes
    if (detection.length > 0) {
      // 009 - walk through all faces detected
      detection.forEach(element => {
        /**
         * 010 - each face element has a expressions attribute
         * for example:
         * neutral: 0.33032259345054626
         * happy: 0.0004914478631690145
         * sad: 0.6230283975601196
         * angry: 0.042668383568525314
         * fearful: 0.000010881130037887488
         * disgusted: 0.003466457361355424
         * surprised: 0.000011861078746733256
         */
        let status = "";
        let valueStatus = 0.0;
        for (const [key, value] of Object.entries(element.expressions)) {
          if (value > valueStatus) {
            status = key
            valueStatus = value;
          }
        }
        // 011 - once we have the highest scored expression (status) we display the right Emoji
        face.innerHTML = statusIcons[status]
      });
    } else {
      console.log("No Faces")
      //face.innerHTML = statusIcons.default;
    }
  }, milliseconds);
}

// 012 - Add a listener once the Video is played
video.addEventListener('playing', () => {
  detectExpression()
})

```

Walking through the code:
- 001 create 2 const to access to the DOM for the video and the emoji div;
- 002 load the needed models previously downloaded. For this PoC we need 2 things: detect face (tinyFaceDetector) and identify the expression (faceExpressionNet). Loading models it takes time. So we are using a promise and then we will call *startvideo* function once the models loading is completed;
- 003 through *getUserMedia* and setting *stream* we access to webcam and show the realtime video in the *video* tag;
- 004 face-api is able to evaluate face detected and assign a kind of scores (from 0 to 1) for each expression: neutral, happy, sad, angry, fearful, disgusted, surprised;
- 005 use a default emoji for default state;
- 006 use *setInterval* to detect periodically the expression (every 500 milliseconds);
- 007 detect faces with expression through *detectAllFaces* method and calling *withFaceExpressions*;
- 008 if no faces is detected *detectAllFaces* will return an empty array (length == 0);
- 009 use *detectAllFaces* (**All**) so an array of faces is returned, so we need to loop through the array;
- 010 each face element has attributes: neutral, happy, sad, angry, fearful, disgusted, surprised. And we track the highest rated expression with *status* and *valueStatus*;
- 011 once we have the highest scored expression (status) we display the right Emoji from the icon map;
- 012 add a listener in order to start detection once the video is played.

## Start your local Web Server

In order to serve your index.html with the assets (js, css, models) you need to start your local web server. You can do it in multiple way. For example if you have PHP installed , in the directory where you have your index.html launch:
```sh
php -S  php -S 127.0.0.1:8081
```
or you can use python 3
```sh
python -m http.server --bind 127.0.0.1 8081
```
Then open your browser and go to URL: [http://127.0.0.1:8081](http://127.0.0.1:8081)


![Surprise!](https://thepracticaldev.s3.amazonaws.com/i/np9bg874b0sqphwxvts3.png)