Face Painting with P5.js
Tags: tutorial, javascript, handsfreejs, creative
Published: 2019-12-02 03:00:48

{% twitter 1201337876933496832 %}

In this tutorial you'll learn everything you need to start painting happy little trees with your face 🌳 This technique uses a "face pointer", which lets you control a pointer with your head and face gestures for clicking and more!

We'll be using the [new Handsfree.js API](https://github.com/handsfreejs/handsfree) to quickly setup our face pointer and [P5.js](https://p5js.org/) to do the painting. Behind the scenes, Handsfree.js is powered by the [Weboji head tracker](https://github.com/jeeliz/jeelizWeboji).

So let's begin!

![https://en.wikipedia.org/wiki/Bob_Ross](https://upload.wikimedia.org/wikipedia/en/7/70/Bob_at_Easel.jpg)

## Setting up our environment

So the first thing we'll want to do is handle dependencies:

```html
<!-- Handsfree.js -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.3/dist/handsfreejs/handsfree.css" />
<script src="https://unpkg.com/handsfree@6.0.3/dist/handsfreejs/handsfree.js"></script>

<!-- Demo dependencies: P5 and lodash -->
<script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.15/lodash.min.js"></script>

<!-- Our P5 Sketch will go in here -->
<div id="canvas-wrap"></div>
```

This gives us a global `Handsfree` class object. The next thing that we do is create an instance of `Handsfree`. We need one instance for every webcam that we plan to use, but each instance can track multiple users ([see config options](https://github.com/handsfreejs/handsfree/wiki/Config)):

```js
const config = {};
handsfree = new Handsfree(config);
```

If at this point we were to run `handsfree.start()` then we would see a red face controlled cursor, along with the debug video feed.

## Adding Functionality

To add functionality you add callbacks (I call them plugins) to the `Handsfree` class object with `Handsfree.use("pluginName", opts)`.

Here `pluginName` can be anything and is there so that we can disable/enable plugins by name with `Handsfree.disable('pluginName')` or access them under the hood with `Handsfree.plugins['pluginName']`.

`opts` can either be a callback function to run on every webcam frame, or it can be an object with the following core properties and methods:

```js
Handsfree.use("pluginName", {
  // Whether to start using this plugin immediately or not
  enabled: true,

  // Called once when the plugin is first used
  // - Use this to initialize stuff
  onUse({ head }) {},

  // Called on every frame loop
  // - Use this as your "game loop"
  // - This is the same as only passing a callback
  onFrame({ head }) {}
});
```

These callbacks pass in the `handsfree` instance, which we usually destructure to get the `handsfree.head` object...these two are equivalent:

```js
Handsfree.use("test1", instance => {
  console.log(instance.head.rotation);
});

Handsfree.use("test2", ({ head }) => {
  console.log(head.rotation);
});
```

Knowing all that, let's define our "P5.facePaint" plugin:

- Setup P5.js and remember get a reference to our canvas
- Capture face gestures on every frame
- Paint and/or change colors

```js
Handsfree.use("p5.facePaint", {
  // Current pointer position
  x: 0,
  y: 0,
  // Last frames pointer position
  lastX: 0,
  lastY: 0,

  // Contains our P5 instance
  p5: null,

  /**
   * Called exactly once when the plugin is first used
   */
  onUse() {
    // Store a reference of our p5 sketch
    this.p5 = new p5(p => {
      const $canvasWrap = document.querySelector("#canvas-wrap");

      // Setup P5 canvas
      p.setup = () => {
        // Create a P5 canvas with the dimensions of our container
        const $canvas = p.createCanvas(
          $canvasWrap.clientWidth,
          $canvasWrap.clientHeight
        );
        $canvas.parent($canvasWrap);
        p.strokeWeight(6);
      };

      // Match canvas size to window
      p.windowResized = () => {
        p.resizeCanvas($canvasWrap.clientWidth, $canvasWrap.clientHeight);
      };
    });
  },

  /**
   * Called on every webcam frame
   */
  onFrame({ head }) {
    // Setup point coordinates
    this.lastX = this.x;
    this.lastY = this.y;
    // @todo: pointer origin should be at center, not corner (fix via CSS?)
    this.x = head.pointer.x + 10;
    this.y = head.pointer.y + 10;

    this.p5.stroke(this.p5.color(strokeColor));

    // Draw lines
    if (head.state.smirk || head.state.smile) {
      this.p5.line(this.x, this.y, this.lastX, this.lastY);
    }

    // Change colors with eyebrows
    if (head.state.browLeftUp) this.updateColor(1);
    else if (head.state.browRightUp) this.updateColor(-1);
  },

  /**
   * Throttled to 4 times a second
   * - Please see source code for this tutorial for the rest of this method
   * @see https://glitch.com/edit/#!/handsfree-face-painting?path=app.js
   */
  updateColor: _.throttle(function(step) {}, 250, { trailing: false })
});
```

## That's all there is to it!

![](https://i.imgur.com/fdL8BAa.gif)

This tutorial quickly went over how to setup a P5 Sketch to work with Handsfree.js. It's still not perfect and I plan to create an official P5.js - Handsfree.js integration soon, but it should be enough to get you going!

Make sure to check out the source for the rest of the code as I omitted some non-Handsfree.js stuff for brevity.

- [Source](https://glitch.com/edit/#!/handsfree-face-painting?path=index.html)
- [Demo](https://handsfree-face-painting.glitch.me)
- [Handsfree.js](https://handsfree.js.org)
- [Twitter @HeyOzRamos](https://twitter.com/heyozramos)

Thanks and have fun 👋