Riffr - Create Photo Montages in the Browser with some ML Magic✨
Tags: atlashackathon22, javascript, machinelearning, opensource
Published: 2022-12-08 23:48:11

## 🚀 What I Built (and What it Does)


Riffr helps you make awesome, awesome montages! 
Okay, that's an oversimplification. Let me clarify how. 

- You're allowed to choose up to 120 images(nice, right?), run face detection models on them 
- Photos with faces detected are then configured around a specific face, zoom in/out levels & a frame rate, that is, how fast the images go by, is set.
- When you're done, the tool compiles these cached images and their configs and wraps it all up together in a neat little `webm` or a `gif` media file. 

Take this `gif`, for example. ([in-app link](https://riffr.tech/view/MoreElonGif-hOZ_H1ztJhbHvXcvf5kgi))
![Elon Montage](https://storage.googleapis.com/riffr-gallery/MoreElonGif-hOZ_H1ztJhbHvXcvf5kgi.gif)

It uses 26 images, cycled at 9 frames per second, and positioned around a specific face(Elon's, in this case).

### 🐱‍👤 Category Submission:

Choose Your Own Adventure

### 🐱‍🏍 App Link

[https://riffr.tech](https://riffr.tech)

### 😏 Background: Some Context
_(What made you decide to build this particular app? What inspired you?)_

Time for a trip down the memory lane. 
Remember this 🔽 video from _wayyy_ back, in 2017?

{% embed https://www.youtube.com/embed/65nfbW-27ps %}
For long, I had been really captivated by how the dedicated Youtuber **Hugo Cornellier**, who took photos in front of a camera every day for 9 years, and ended up making a time-lapse collection of over 2500 photos, placed perfectly around himself. This video spoke to me, as it probably did to you. It showed a person's journey - of growth, of being a kid, of going through phases, and finally, finding love.

However, it turns out, capturing this together was a painstakingly slow and difficult process. Aligning along a specific face, resizing images, and compiling these to make a meaningful video. It took Hugo 50+ hours just to make this!

Being inspired by his pain, I decided to lend a hand and put out a tool that can automate all this configuration, detect and cycle between faces, and generate a video with it all!

**Psst...**With this app, a user doesn't even have to click photos repeatedly at the same place or in the same position - the ML model adjusts all sorts of photos together 🤭

Sounds easy enough to implement, right? 
Spoiler Alert: It was _not_ easy at all.

![Ummmm](https://media.giphy.com/media/d5Fbtoj47ROk0J9IgW/giphy.gif)

### 🛠 How I Built It
_(How did you utilize MongoDB Atlas? Did you learn something new along the way? Pick up a new skill?)_

Since my first implementation (nearer to the beginning of the Atlas hackathon) was focused on preserving user privacy, I went for a pre-trained ML model that was loaded right into the browser, and compiled using the native `MediaRecorder` Web API. Additionally, I learned how to cache images in the browser using `Web Workers` and draw them blazing-fast onto the canvas with `ImageBitmap` object.

**_Note_** - For more details, check out my write-up on that earlier implementation that I published on Devpost ([here](https://devpost.com/software/riffr)) & submitted to another hackathon.

Nevertheless, the limitations caught up, and while various bug fixes ensured good quality (using `VP8` media codecs) and speed, on mobile devices, it was just too much sometimes, and they'd just freeze due to large memory & computing requirements and start burning up!

![fire!](https://media.giphy.com/media/nrXif9YExO9EI/giphy.gif)

After a while, I decided to overcome this by using the power of `Tensorflow-Node` in a custom Node.js backend, one that I could fully manage and let users share, upload and detect without worries about their phones. The compilation is still done on the front end, though, since it is easier there, in both `gif` & `webm` formats. Even though it might seem like a privacy-performance trade-off, I have structured the application in such a way with MongoDB that it's really easy to maintain privacy.

Plus, now it's as easy as toggling a button to switch between in-browser and on-server ML detection!

Lastly, the VueUse library made it so easier to tackle memory heap allocation debugging, QR Code generation, color picking, web workers etc.

#### 💪 Using MongoDB Atlas

I learned the Mongoose ODM to integrate with Atlas, and used it as a back end database (with possible future search functionalities) to create 2 collections, `Montages` & `Users`.

![MongoDB Atlas Collections](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o4ze97p2ljzvc33gq9h8.png)

The server generates unique, secret urls that are inherently public but as secure & private as `uuid` using the `nanoid` module, and storea them publicly in both the database and storage solution. This way, fetching is easier, data can be aggregated in one place or distributed accordingly, and indexes for search purposes can be generated, too!


### 📸 Screenshots & Details

#### Signup/Signin & Onboarding Processes
<table>
  <tr>
     <td>Landing Page with Signup/Signin</td>
     <td>Onboarding Intro</td>
  </tr>
  <tr>
    <td valign="top"><img src = "https://i.ibb.co/tcX17yy/Screenshot-2022-12-09-041341.png" width=500 /></td>
    <td valign="top"><img src = "https://i.ibb.co/chqQwZL/Screenshot-2022-12-09-042432.png" width=500 /></td>

  </tr>
  <tr>
     <td>Onboarding Avatar & Email Verification</td>
     <td>Onboarding Details</td>
  </tr>

  <tr>
    <td valign="top"><img src = "https://i.ibb.co/0YqMKKv/Screenshot-2022-12-09-042537.png" width=500 /></td>
    <td valign="top"><img src = "https://i.ibb.co/0DwLx5r/Screenshot-2022-12-09-042607.png" width=500 /></td>
  </tr>
</table>

#### Pick, Edit & Compile

<table>
  <tr>
     <td>User Aggregated Dashboard</td>
     <td>Add your pictures, choose output & model</td>
  </tr>
  <tr>
    <td valign="top"><img src = "https://i.ibb.co/8zgKLpv/Screenshot-2022-12-09-042643.png" width=500 /></td>
    <td valign="top"><img src = "https://i.ibb.co/QcP1FXh/Screenshot-2022-12-09-042727.png" width=500 /></td>

  </tr>
  <tr>
     <td>Select the right faces, frame rate, zoom, background</td>
     <td>Compile, and Voila! Your montage is ready</td>
  </tr>

  <tr>
    <td valign="top"><img src = "https://i.ibb.co/nnv8R5G/Screenshot-2022-12-09-042854.png" width=500 /></td>
    <td valign="top"><img src = "https://i.ibb.co/48x8tyf/Screenshot-2022-12-09-042947.png" width=500 /></td>
  </tr>
</table>

#### Viewer Page (with Publicly Accessible Links)

![Viewer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vu0jzlnl6knl7tsveclh.png)

### 👨‍💻 Tech Stack

Basically a MEVN stack web application. However, there are additionally a large number of APIs used, too.

- Front End - [Quasar Framework](https://quasar.dev/) built on top of Vue 3.
- Back End - [Node.js](https://nodejs.org/), [Express.js](https://expressjs.com/) & [MongoDB Atlas](https://www.mongodb.com/atlas/database)
- Important APIs - [ml5](https://ml5js.org/) for in-browser detection, [face-api](https://github.com/vladmandic/face-api) that uses `tensorflow-node` to accelerate on-server detection. [VueUse](https://vueuse.org/) for a bunch of useful component tools like the `QR Code` generator. Yahoo's [Gifshot](https://github.com/yahoo/gifshot) for creating `gif` files in-browser etc.
- Storage Vendor - [Google Cloud Storage](https://console.cloud.google.com/storage/)
- Hosting - [Azure](https://portal.azure.com/) Virtual Machine for the `Express` server, and [Vercel](https://vercel.app/) for `Quasar` build.
- Transactional - [Mailgun](https://www.mailgun.com/) to send verification emails

### 🔗 Link to Source Code
{% embed https://github.com/holy-script/riffr %}

### 📃 Permissive License
`MIT License`

### 🤫 Additional Resources/Info

Riffr's original submission page (on [Devpost](https://devpost.com/software/riffr)) that won 1st spot 🏆 in an MLH Weekend Hackathon.