Handsfree.js - A web based Face Pointer
Tags: javascript, computervision, handsfreejs, tutorial
Published: 2019-11-08 04:39:48

{% twitter 1191514501859831809 %}

Handsfree.js is a small wrapper library around web based computer vision libraries that I'm working on for the purpose of helping you interact with web pages handsfree. Currently it only handles head tracking through [Jeeliz Weboji](https://github.com/jeeliz/jeelizWeboji), but I also plan to add eye tracking, hand tracking, voice, and full body pose estimation over [my next 100 days of code](https://twitter.com/heyozramos).

I also plan on adding integrations for popular libraries like [P5.js](https://p5js.org/), [Babylon.js](https://www.babylonjs.com/), [A-Frame](https://aframe.io/), and others!

But in this post, I'll just focus on how to get a simple Handsfree.js project going which involves three steps:

- Requiring dependencies
- Creating a `handsfree` instance
- Adding a plugin (a callback that runs on every video inference frame)

At the end, you'll have a red face controlled pointer like in the following:

{% twitter 1191943329023512576 %}

## Adding a handsfree pointer

Because everything is already bundled for you, all you need to do is include the Handsfree.js JavaScript and Stylesheet in your DOM:

```html
<!-- Require dependencies -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
<script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>
```

This adds the `Handsfree` class to your page, along with basic styles for the pointer. The next step is to create an instance of `Handsfree`:

```js
const config = {}
const handsfree = new Handsfree(config)
```

You'll need one instance for each camera that you plan to use (if you have multiple cameras), but each instance can track multiple users. See the [documentation on the config object](https://github.com/handsfreejs/handsfree) for the defaults and other settings you can use.

And that's it! To start and stop tracking, use `handsfree.start()` and `handsfree.stop()`.

## Adding functionality, and using the pointer values

On each frame, your `handsfree` instance will have several properties that contain useful information:

```js
// The x coordinate of the pointer on the screen
handsfee.head.pointer.x

// The y coordinate of the pointer on the screen (from 0 at the top)
handsfee.head.pointer.y

// The pointer element
handsfree.head.pointer.$el

// The pointer state ("", "mouseDown", "mouseDrag", "mouseUp")
handsfree.head.pointer.state

// The head position [x, y, scale]
handsfree.head.translation

// The head rotation [pitch, yaw, roll]
handsfree.head.rotation

// Head Morphs (how raised the eyebrows are, how smiley your smiling to each side, etc)
// @see https://github.com/handsfreejs/handsfree/wiki/Head
handsfree.head.morphs
```

The `Handsfree` class has a global loop for all instances which you can hook into with the `use` method:

```js
// Create a simple plugin that displays pointer values on every frame
Handsfree.use('consoleLogger', (instance) => {
  console.log(instance.head.morphs)
})

// Same plugin, but with destructuring
Handsfree.use('consoleLogger', ({head}) => {
  console.log(head.morphs)
})
```

These are called plugins, where `"consoleLogger"` is the name of the plugin and `instance` is the handsfree instance running the plugin (`handsfree = new Handsfree()`). Adding multiple plugins with the same name overwrites the previous plugin, and to disable a plugin you can call `handsfree.stop()`.

Handsfree.js ships with a few plugins - "head.click" and "head.vertScroll" - which add clicking functionality (with a smile gesture) and scrolling as in this tweet:

{% twitter 1192302275344994304 %}

## A complete example

```html
<!DOCTYPE html>
<head>
  <!-- Require dependencies -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
  <script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>
  
  <style>
    button {font-size: 24px; padding: 20px}
  </style>
</head>
<body>
  <!-- Let's always ask the user to start -->
  <button onclick="handsfree.start()">Start Webcam</button>
  <button onclick="handsfree.stop()">Stop Webcam</button>
  
  <script>
    // Create a new instance. Use one instance for each camera
    const handsfree = new Handsfree({})

    // Create a simple plugin that displays pointer values on every frame
    Handsfree.use('consoleLogger', ({head}) => {
      console.log(head.rotation)
    })
  </script>
</body>
```

[Run the above code on Glitch.](https://glitch.com/edit/#!/handsfree-minimal)

## Next Steps

I hope this gave you a good introduction into Handsfree.js. In this post I covered how to setup a basic pointer, and in future posts we'll be going over:

- How to use head morphs (eyebrows, smiles, kissy faces, and more)
- Get the users head pose (yaw, pitch, roll)
- Multiplayer support
- Complex plugins
- Integrations with popular frameworks
- and more

Follow me on [Twitter at @HeyOzRamos](https://twitter.com/heyozramos) if you'd like to follow along on my 100DaysofCode. Thanks for reading!
Oz

## Updates
- 11/9 - Added references to a few new properties: `handsfree.head.rotation`, `handsfree.head.morphs`. New docs at: https://github.com/handsfreejs/handsfree/wiki
- 11/23 - Updated to reflect new v6 API