Face off : Three(.js)
Tags: photogrammetry, prnet, netlify, threejs
Published: 2019-09-23 22:30:49

---
title: "Face off : Three(.js)"
published: true
description: Using PRNet to generate a 3d mesh from photos and then viewing them online using threejs 
tags: photogrammetry, prnet, netlify, threejs
---

This is a cross-post from https://www.webpusher.ie/2019/09/23/face-off

I've been interested in [photogrammetry](https://en.wikipedia.org/wiki/Photogrammetry) which involves lots of photos, but what is the least number of photos required? Well, it turns out that all you need is one photo.

In this article I'll quickly run through how to generate a 3d mesh from a photograph, and then use [netlify](https://www.netlify.com/) to publish a [threejs](https://threejs.org/) based model viewer to see the final result. The process is entirely manual at the moment!

## How to create a 3d mesh?

After some failed experiments with mobile photogrammetry apps I decided it was too difficult to easily capture many photos.

![bad 3d construction of a guitar using photogrammetry](https://thepracticaldev.s3.amazonaws.com/i/uk3zzias690uulh7tis4.png)

### OpenCV

I then tried [OpenCV Stereo SGBM](https://docs.opencv.org/3.4/d1/d9f/classcv_1_1stereo_1_1StereoBinarySGBM.html) which uses stereo photos, compares the differences and uses geometry, and some advanced techniques, to estimate a depth map, then colour it uses the original image.

This was very promising when using the example photos, but despite spending a lot of time on it, I found it impossible to get even barely passable results. The image below shows headphones resting on an orange book - and that is the best angle to view it from!

![bad point cloud of Headphones and book](https://thepracticaldev.s3.amazonaws.com/i/haa4cc7dg7yi5a2tiv8s.png)


This is definitely something I will come back to. I didn't try it at the time, but [there is a tool to help tweak the parameters](https://github.com/hitimo/opencv-disparity-map-tuner). The other disadvantage is that building OpenCV is an arduous process.

### Machine Learning to the rescue!

I used the popular PRNet CNN (convolutional neural net) to generate a wire mesh, which met with immediate success! 

![gurning face and 3d model of the same gurning face](https://thepracticaldev.s3.amazonaws.com/i/b7r8hn25k2iz0d89s3ow.png)


[There is an accessible(ish) explanation](https://medium.com/@hyprsense/bridging-the-academia-gap-an-implementation-of-prnet-training-9fa035c27702) of how the process works.

### Is this a fake?

More like a deep fake!

PRNet is trained to generate a [UV map](https://en.wikipedia.org/wiki/UV_mapping), which encodes both volume and shape of a 3d object. Three important elements can be generated from this UV map

- sparse feature point prediction - eyebrows, eyes, nose, mouth, lips, and chin
- dense feature point prediction
- full 3-dimensional mesh model, with colors from the input image.

### Show me the code!

It's not just an academic paper [there is fully working example code](https://github.com/YadiraF/PRNet)

It takes a little bit of work to get this running I'd recommend using my [Pyenv configuration advice from a previous article](https://www.webpusher.ie/2018/09/19/python-dependency-hell-no).

I stuck to the requirements to get this working - Python 2.7 - so I didn't investigate if more recent versions of Python would work. If you do bump your Python version be aware that as of publishing (September 2019) dependencies of TensorFlow limit Python to version 3.6.

```bash
$ pyenv virtualenv 2.7.14 prnet
$ pyenv local prnet
```

You'll also need to install `dlib`.

```bash
$ pip install --upgrade dlib
```

I'd recommend upgrading `pip` and `setuptools` to the latest version before you install the project dependencies

```bash
$ pip install --upgrade pip setuptools
```

Now you can install the dependencies

```bash
$ pip install -r requirements.txt
```

### Generating the mesh

You'll need some images saved in a known directory. Once you have that, simply run:

```bash
$ python demo.py -i {the directory holding images with faces} -o {output directory for the obj files}
```

You'll probably see lots of warnings (I did) which would likely be reduced or removed by upgrading your Python version, but I'm not doing to until later in the project!

In your output folder you should see a `.obj` file for each image you provided. Note, you will only see one face per image - there is no automatic support for multiple faces.

## Viewing the output

I used the free and capable [MeshLab](http://www.meshlab.net/) to immediately open and view the models - I laughed at my stupid face spinning before me!

Even though it was not a goal of my mystery side-project, I wanted to leverage my webdev skills to put this online for all to see - everyone should be laughing at my stupid spinning face!

### Threejs and Netlify to the rescue!

I use Netlify to publish my Gatsby based blog, so I knew it was going to do the heavy-lifting of deploying my new 3d monster gallery.

I didn't want to be messing about with bundling the 'app' so I just wrote a little shell script to copy the `threejs` library and associated support code from the `node_modules` directory to the `src` folder where I could import it into my es6 modules.

The netlify build settings I used are:

- Repository : github.com/mlennox/three-dee-mee
- Base directory : Not set
- Build command : sh prepare.sh
- Publish directory : src

The [full, working code is on github](https://github.com/mlennox/three-dee-mee) and the actual working demo is available at https://thirsty-hypatia-f27848.netlify.com/ and looks like this (it may take a few seconds to initialise)

![Screengrab of model viewer demo showing 3d model and list of thumbnails to click](https://thepracticaldev.s3.amazonaws.com/i/c06awtku2bwcsa1u17yp.png)

Just click the gurning visages in the row of thumbnails to load the associated three-d mesh, and you can control the hideous visage by clicking, dragging, and scrolling to zoom.