TensorFlow + Next.js + TypeScript: Remove background and add virtual background image with web camera
Tags: tensorflow, machinelearning, nextjs, typescript
Published: 2021-07-26 03:01:06

Hello guys,

I have developed an application with [BodyPix](https://github.com/tensorflow/tfjs-models/tree/master/body-pix), that removes an original background and add a new virtual background image.

[![Image from Gyazo](https://i.gyazo.com/d38a9a41d834d767588917846bce4d7a.gif)](https://gyazo.com/d38a9a41d834d767588917846bce4d7a)

In this article, I will explain how to develop this application.

DEMO→https://travel-app-three.vercel.app/
github→https://github.com/yuikoito/tensorflow-bodypix-sample

## Setup Next.js application and install react-webcam

```
$ yarn create next-app <app-name>
$ cd <app-name>
$ touch tsconfig.json
$ yarn add --dev typescript @types/react
```

Then, rename index.js and _app.js to index.tsx, _app.tsx.

Install webcam.

```
$ yarn add react-webcam @types/react-webcam
```

Now, we are ready to develop :)


## Install TensorFlow.js

When using TensorFlow.js with TypeScript, you need to be a little careful.

I wrote in [another article](https://dev.to/yuikoito/face-detection-by-using-tensorflow-react-typescript-3dn5) as well, but you shouldn't do just ``yarn add @tensorflow/tfjs`` when you use TypeScript, otherwise you would get a type error.

Then, install like as following.

```
$ yarn add @tensorflow-models/body-pix @tensorflow/tfjs-core @tensorflow/tfjs-converter @tensorflow/tfjs-backend-webgl
```

In this case, we are also using the [body-pix model](https://github.com/tensorflow/tfjs-models/tree/master/body-pix), so install it as well.

## Set up to use bodyPix

As stated in [the official documentation](https://github.com/tensorflow/tfjs-models/tree/master/body-pix), it is very simple to use.
All you have to do is 1) import bodyPix, 2) load it, and 3) when the loading is complete, put the image data you want to analyze into an argument in the ``segmentPerson`` function.

Now, we're writing it in Next.js, so the code is like this.

```react
// first, import all you need
import { useRef, useState, useEffect } from "react";
import Head from "next/head";
import "@tensorflow/tfjs-core";
import "@tensorflow/tfjs-converter";
import "@tensorflow/tfjs-backend-webgl";
import styles from "../styles/Home.module.scss";
import * as bodyPix from "@tensorflow-models/body-pix";
import Webcam from "react-webcam";

function Home() {
  const canvasRef = useRef<HTMLCanvasElement>(null);
  const webcamRef = useRef<Webcam>(null);
  // Manage the state of bodypixnet with useState
  const [bodypixnet, setBodypixnet] = useState<bodyPix.BodyPix>();

  // Run only when the page is first loaded
  useEffect(() => {
    bodyPix.load().then((net: bodyPix.BodyPix) => {
      setBodypixnet(net);
    });
  }, []);

  const drawimage = async (
    webcam: HTMLVideoElement
  ) => {
    const segmentation = await bodypixnet.segmentPerson(webcam);
    console.log(segmentation);
  };

  const clickHandler = async () => {
    const webcam = webcamRef.current.video as HTMLVideoElement;
    const canvas = canvasRef.current;
    // Make the canvas, webcam, and video size all the same size.
    webcam.width = canvas.width = webcam.videoWidth;
    webcam.height = canvas.height = webcam.videoHeight;
    const context = canvas.getContext("2d");
    context.clearRect(0, 0, canvas.width, canvas.height);
    // If it is clicked before bodypixnet is set, it will cause an error, so just in case.
    if (bodypixnet) {
      drawimage(webcam);
    }
  };
  return (
    <div className={styles.container}>
      <Head>
        <title>Travel App</title>
        <meta name="description" content="Generated by create next app" />
        <link rel="icon" href="/static/logo.jpg" />
      </Head>
      <header className={styles.header}>
        <h1 className={styles.title}>Title</h1>
      </header>
      <main className={styles.main}>
        <div className={styles.videoContainer}>
          <Webcam audio={false} ref={webcamRef} className={styles.video} />
          <canvas ref={canvasRef} className={styles.canvas} />
        </div>
        <div className={styles.right}>
          <h4 className={styles.title}>{t.select}</h4>
          <div className={styles.buttons}>
            <button onClick={clickHandler}>
              Button
            </button>
          </div>
        </div>
      </main>
    </div>
  );
}

export default Home;
```

Let's make sure that the bodypix is working correctly.

When the button is clicked, all elements of the webcam, canvas, and video data are set in the same size.

In ``drawimage``, we can parse the video data (webcam) by putting it in the argument of ``bodypixnet.segmentPerson``. (The following part)

```react
const segmentation = await bodypixnet.segmentPerson(webcam);
```

If the segmentation is correctly output to the log, everything is fine.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/670446/6208d7b4-b7ba-3da8-fdcd-59eab04c83fb.png)

By the way, if you just want to blur the background or separate colors for people and others, you can easily do so as follows.

```react
const coloredPartImage = bodyPix.toMask(segmentation);
const opacity = 0.7;
const flipHorizontal = false;
const maskBlurAmount = 0;
const canvas = document.getElementById('canvas');
// Draw the mask image on top of the original image onto a canvas.
// The colored part image will be drawn semi-transparent, with an opacity of
// 0.7, allowing for the original image to be visible under.
bodyPix.drawMask(
    canvas, img, coloredPartImage, opacity, maskBlurAmount,
    flipHorizontal);
```

If need, read https://github.com/tensorflow/tfjs-models/tree/master/body-pix#output-visualization-utility-functions.

With ``bodyPix.drawMask``, you can easily specify the transparency, blur the border, flip the left and right sides, and so on.

Here, we will not use ``bodyPix.drawMask``, but use ``drawImage()`` directly on canvas.

## Remove the background to make a virtual background

Although I wrote **remove** background, I am not actually hollowing out the background to create a transparent image.

I've been doing some research, and it seems that if we want to completely remove the background, we may need to use WebGL.

Reference→[Using BodyPix segmentation in a WebGL shader](https://jameshfisher.com/2020/09/24/using-bodypix-segmentation-in-a-webgl-shader/)

Therefore, I decided to use ``destination-out`` of canvas.
(``xor`` is also possible.)

As you can see in the figure below, ``destination-out`` remove overlapping areas.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/670446/02e1ebff-af50-4a98-4d8a-937b6cb52074.png)

So, if we make a temporary canvas element and put the image obtained by ``bodyPix.toMask()`` on it, only the mask area will be removed and we can achieve the virtual background.

You can also see in the log that the contents of ``bodyPix.toMask()`` are stored as ImageData.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/670446/3e7caa20-fc9d-65c3-14ab-ea8f70d5721e.png)

OK, now it's time to write a code!

```react
  const drawimage = async (
    webcam: HTMLVideoElement,
    context: CanvasRenderingContext2D,
    canvas: HTMLCanvasElement
  ) => {
    // create tempCanvas
    const tempCanvas = document.createElement("canvas");
    tempCanvas.width = webcam.videoWidth;
    tempCanvas.height = webcam.videoHeight;
    const tempCtx = tempCanvas.getContext("2d");
    const segmentation = await bodypixnet.segmentPerson(webcam);
    const mask = bodyPix.toMask(segmentation);
    (async function drawMask() {
      requestAnimationFrame(drawMask);
      // draw mask on tempCanvas
      const segmentation = await bodypixnet.segmentPerson(webcam);
      const mask = bodyPix.toMask(segmentation);
      tempCtx.putImageData(mask, 0, 0);
      // draw original image
      context.drawImage(webcam, 0, 0, canvas.width, canvas.height);
      // use destination-out, then only masked area will be removed
      context.save();
      context.globalCompositeOperation = "destination-out";
      context.drawImage(tempCanvas, 0, 0, canvas.width, canvas.height);
      context.restore();
    })();
  };
```

Yay! Now that the background is transparent, then we can add the background image to the canvas element.

You can add the background image directly by using ``canvas.style.backgroundImage``, but here I recommend you to add a class name because there is a possibility images may not load instantly.

So, I wrote like this.

```react
  const clickHandler = async (className: string) => {
    const webcam = webcamRef.current.video as HTMLVideoElement;
    const canvas = canvasRef.current;
    webcam.width = canvas.width = webcam.videoWidth;
    webcam.height = canvas.height = webcam.videoHeight;
    const context = canvas.getContext("2d");
    context.clearRect(0, 0, canvas.width, canvas.height);
    canvas.classList.add(className);
    if (bodypixnet) {
      drawimage(webcam, context, canvas);
    }
  };
```

css is as following.

```css
.turky {
  background-image: url(../assets/turky.jpg);
  background-size: cover;
}
```

Then, when the ``turky`` class is added, ``/assets/turky.jpg`` will be placed as the background image.

FYI, if you need to prepare several class names as I did, you need to remove the class name before you add a new one.

```react
    if (prevClassName) {
      canvas.classList.remove(prevClassName);
      setPrevClassName(className);
    } else {
      setPrevClassName(className);
    }
    canvas.classList.add(className);
```

Then finish!

By the way, the background images are all I took before.
I hope to be able to travel soon!

This is a kind of advertisement, but I copied and pasted in [ui-components](https://www.ui-components.com/Button), which I developed for the button layout.
It's convenient!

## That's it!

This article is the tenth week of trying to write at least one article every week.

If you'd like, please take a look at my previous weekly posts!
See you soon!


- [Face detection with Tensorflow【React + TypeScript】](https://dev.to/yuikoito/face-detection-by-using-tensorflow-react-typescript-3dn5)
- [UI Components website Released!](https://dev.to/yuikoito/ui-components-website-released-2g8e)
- [I made 18 UI components for all developers](https://dev.to/yuikoito/i-made-18-ui-components-for-all-developers-17l6)
- [Image Transformation: Convert pictures to add styles from famous paintings](https://dev.to/yuikoito/image-transformation-convert-pictures-to-add-styles-from-famous-paintings-24ml)
- [Developed an app to transcribe and translate from images](https://dev.to/yuikoito/developed-an-app-to-transcribe-and-translate-from-images-32gp)
- [Generate Open Graph images with Next.js and TypeScript on Vercel](https://dev.to/yuikoito/generate-open-graph-images-with-next-js-and-typescript-on-vercel-110j)
- [OpenCV in Lambda: Created an API to convert images to Pokémon ASCII art style using AWS and OpenCV](https://dev.to/yuikoito/opencv-in-lambda-created-an-api-to-convert-images-to-pokemon-ascii-art-style-using-aws-and-opencv-287l)
- [Developed a web app to convert images to Pokémon ASCII art style](https://dev.to/yuikoito/developed-a-web-app-to-convert-images-to-pokemon-ascii-art-style-mb7)
- [Using OpenCV: Developed a web app to convert images to manga style](https://dev.to/yuikoito/using-opencv-developed-a-web-app-to-convert-images-to-manga-style-23p0)


## Contact

Please send me a message if you want to offer a job or ask me something.

yuiko.dev@gmail.com
https://twitter.com/yui_active