Instagram SparkAR "clone yourself" segmentation effect with Handsfree.js and p5.js
Tags: tutorial, p5, handsfreejs, computervision
Published: 2020-01-22 18:53:42

![A clip of a million clones of Oz Ramos all acting as one](https://i.imgur.com/4YifsWa.gif)

Demo: https://editor.p5js.org/heyozramos/sketches/LVsjLSPt7

---

A while back I came across a really fun tweet by @Cassidoo on Twitter

{% tweet 1153738731188187141 %}

At the end of the clip, there's a scene where a million copies of her are singing together. It's a really cool effect and in this first tutorial in a series on Segmentation Effects, I'll show you how to recreate it with JavaScript!

## Our tools

We'll be using:

- [p5.js](https://p5js.org/) in global mode. p5.js is a beginner-friendly creative coding library
- [Google's Bodypix](https://blog.tensorflow.org/2019/11/updated-bodypix-2.html). Bodypix is a segmentation and pose estimation computer vision library powered by TensorFlow. We'll initialize BodyPix through Handsfree.js
- [Handsfree.js](https://browsehandsfree.com/code/) is my library for interacting with web pages handsfree

```html
<head>
    <script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script>
    <script src="https://unpkg.com/handsfree@6.1.7/dist/handsfree.js"></script>
    <link rel="stylesheet" type="text/css" href="https://unpkg.com/handsfree@6.1.7/dist/handsfree.css">
</head>
```

## Instantiating Handsfree.js

Handsfree.js is a library that wraps popular computer vision models like [Jeeliz Weboji](https://github.com/jeeliz/jeelizWeboji) (head tracking), [Google's BodyPix](https://blog.tensorflow.org/2019/11/updated-bodypix-2.html) (segmentation and pose estimation), and soon, eye tracking through [WebGazer](https://webgazer.cs.brown.edu/) and Handtracking through [Handtrack.js](https://github.com/victordibia/handtrack.js/). I originally started it as an assistive tech tool to help people browse the web handsfree through face gestures, but it also doubles as a controller for games, [Algoraves](https://www.youtube.com/watch?v=h340aNznHnM), and other creative coding projects!

{% tweet 1191514501859831809 %}

So to begin, we need to instantiate Handsfree.js. We instantiate once for every camera that we want to use (if you have a multi-cam setup or using a mobile device with front/back webcams). Instantiation looks like this:

```js
/**
 * This method gets called once automatically by p5.js
 */
function setup () {
  // Instantiate Handsfree. "H"andsfree is the class, "h"andsfree is the instance
  handsfree = new Handsfree({
    autostart: true,
    models: {
      head: {enabled: false},
      bodypix: {
        enabled: true,
        
        // Uncomment this if it runs slow for you
        // modelConfig: {
        //   architecture: 'MobileNetV1',
        //   outputStride: 8,
        //   multiplier: 0.75,
        //   quantBytes: 2
        // }
      }
    }
  })

  // Disable default Handsfree.js plugins
  Handsfree.disableAll()
}
```

## Creating a buffer canvas

The next thing we'll need to do is create a "buffer" canvas. This canvas will contain the cutout, which we'll then paste into the p5 canvas a bunch of times:

```js
function setup () {
  // ...

  // Create a canvas that'll contain our segmentation
  $buffer = document.createElement('canvas')
  $buffer.style.display = 'none'
  bufferCtx = $buffer.getContext('2d')
  document.body.appendChild($buffer)

  // This part's important, match the buffer dimensions to the webcam feed
  $buffer.width = handsfree.debugger.video.width
  $buffer.height = handsfree.debugger.video.height
  widthHeightRatio = $buffer.width / $buffer.height

  // Create a slider to adjust the number of clones
  slider = createSlider(3, 12, 4, 1)

  // Finally, create the p5 canvas
  renderer = createCanvas(600, 600)
  $canvas = renderer.canvas
  canvasCtx = $canvas.getContext('2d')
}
```

## Creating a mask

![](https://1.bp.blogspot.com/-Xg2-vlsCYGw/Xc7knfZ34FI/AAAAAAAAA7o/gNGck4piZ_IStvxbzCZlDVqRm-ntDwObgCLcBGAsYHQ/s1600/1.gif)

(GIF source: https://blog.tensorflow.org/2019/11/updated-bodypix-2.html)

We'll be using BodyPix as a kind of "smart green screen" that's able to remove all the background pixels. We'll do this in a method called `draw` that is run automatically by p5 every few milliseconds:

```js
function draw() {
  background(220);
  
  // Just for convenience
  const model = handsfree.model
  const body = handsfree.body

  // Only paint when we have data
  if (body.data) {
    // Create a mask, with all non segmented pixels as magenta
    mask = model.bodypix.sdk.toMask(
      body.data,
      {r: 0, g: 0, b: 0, a: 0},      // foreground
      {r: 255, g: 0, b: 255, a: 255} // background
    )
    
    // Paint the mask into a buffer canvas
    model.bodypix.sdk.drawMask(
      $buffer,
      handsfree.debugger.video,
      mask,
      1, 0, 0
    )

    // Make all magenta pixels in the buffer mask transparent
    let imageData = bufferCtx.getImageData(0, 0, $buffer.width, $buffer.height)
    for (let i = 0; i < imageData.data.length; i += 4) {
      if (imageData.data[i] === 255 && imageData.data[i + 1] === 0 && imageData.data[i + 2] === 255) {
        imageData.data[i + 3] = 0
      }
    }
    bufferCtx.putImageData(imageData, 0, 0)

    // Dimensions of each mask
    w = $buffer.width / slider.value()
    h = w * widthHeightRatio

    // Paste the mask a bunch of times
    for (row = 0; row < slider.value() * 3; row++) {
      for (col = 0; col < slider.value(); col++) {
        // Stagger every other row
        x = col * w - (w / 2) * (row % 2)
        y = row * h / 3 - h / 2 * (1 + row / 10)

        canvasCtx.drawImage($buffer, x, y, w, h)
      }
    }
  }
}
```

---

On each webcam frame, Handsfree.js captures segmentation data into `handsfree.body.data` which we'll use to create the mask.

```js
    // Create a mask, with all background pixels as magenta (255, 0, 255)
    mask = model.bodypix.sdk.toMask(
      body.data,
      {r: 0, g: 0, b: 0, a: 0},      // foreground
      {r: 255, g: 0, b: 255, a: 255} // background
    )

    // Paint the mask into a buffer canvas
    model.bodypix.sdk.drawMask(
      $buffer,
      handsfree.debugger.video,
      mask,
      1, 0, 0
    )

```

![](https://i.imgur.com/tOOH2h4.jpg)

Next, we'll go through each pixel and make any magenta pixels transparent:

```js
    // Make all magenta pixels in the buffer mask transparent
    let imageData = bufferCtx.getImageData(0, 0, $buffer.width, $buffer.height)
    for (let i = 0; i < imageData.data.length; i += 4) {
      if (imageData.data[i] === 255 && imageData.data[i + 1] === 0 && imageData.data[i + 2] === 255) {
        imageData.data[i + 3] = 0
      }
    }
    bufferCtx.putImageData(imageData, 0, 0)
```

Finally, we paste the buffer into the p5 canvas a bunch of times:

```js
    // Dimensions of each mask
    w = $buffer.width / slider.value()
    h = w * widthHeightRatio

    // Paste the mask a bunch of times
    for (row = 0; row < slider.value() * 3; row++) {
      for (col = 0; col < slider.value(); col++) {
        // Stagger every other row
        x = col * w - (w / 2) * (row % 2)
        y = row * h / 3 - h / 2 * (1 + row / 10)

        canvasCtx.drawImage($buffer, x, y, w, h)
      }
    }

```

## And that's all there is to it!

Let me know in the comments if anything is unclear. I hope you enjoy playing around with this demo, it's the first in a set of tutorials on creating [Instagram SparkAR](https://sparkar.facebook.com/ar-studio/)-like filters for the web with p5.js.

- [Twitter @HeyOzRamos](https://twitter.com/heyozramos)
- [Handsfree.js](https://handsfree.js.org)
- [BodyPix](https://blog.tensorflow.org/2019/11/updated-bodypix-2.html)
- [P5.js](https://p5js.org/)

---

![A clip of a million clones of Oz Ramos all acting as one](https://i.imgur.com/4YifsWa.gif)

Demo: https://editor.p5js.org/heyozramos/sketches/LVsjLSPt7