SignLanguage - Learn ASL Practically - (MongoDB Atlas Hackathon 2022 Submission)
Tags: atlashackathon22, machinelearning, mongodb, javascript
Published: 2022-12-08 18:16:50

## What I built
**SignLanguage** now **GestureAcademy** is a platform where users can practically learn American Sign Language using machine learning and access videos for over 20,000+ ASL phrases.

![signLanguage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9kxwd6qf4zfy5kworzcu.gif)



### Category Submission:  Search No More


### App Link 🎉🎉🥳
https://gestureacademy.webdrip.in/


### App Previews & Features

#### Homepage

The homepage displays the entire application preview. Users can look up terms for videos and alphabets, as well as play games.

![Signlanguage Homepage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b0nqjldx8mjukffs9g41.png)

#### Phrases Dictionary

Signlanguage includes around 20,000 phrases from which users can learn ASL.

![Phrases](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/upd3dcbs0brsti8fxi2c.png)

![open video](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/96s3xbnddbpumjpmqgfc.png)

#### Lightning Fast Fuzzy Search

Users can search for any video from 20,000+ video using the fuzzy search feature implemented using MongoDB Atlas search in the ASL phrases dictionary.


![Fuzzy Search](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/im8k3fhse0qeang5bi7j.png)

![Fuzzy Search play video](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1fwna63j350pgdergcia.png)

The video below shows how quickly MongoDB can search through over 20,000 video documents.

{% vimeo 779291533 %}


## Game Of ASL

Users can play games that have been created using mediapipe and tensorflow machine learning libraries to help them learn and validate their learning.

### Learn Numbers

Learn numbers from 0-10 in chronological order with the help of artificial intelligence.

{% vimeo 779290403 %}

### Random Numbers

Show your hands on screen and AI will try to predict the numbers from 0-10 based on your hand signs.

{% vimeo 779290243 %}
 

### Link to Source Code

{% embed https://github.com/Narottam04/SignLanguage %}

### Permissive License
`MIT License`

## Background

Developing software applications or products that solve real world problem statements has always intrigued me. When I started learning ML and explored it's possibilities I thought of making a project that would be a part of something bigger than itself i.e. helping out community and reaching people in an easily accessible manner. This was my motivation for developing this app SignLanguage.

YouTube is a great source of knowledge, but its algorithm doesn't promote short videos or videos that entice users. SignLanguage solves this problem by collecting these resources and making them easily accessible through our webapp. This webapp has around 20,000+ curated video phrases and features like fuzzy Search which can help users to learn American Sign Language(ASL) with ease.

SignLanguage also has a number of games that users can play to practice their ASL fundamentals while having fun.

### Tech Stack & Libraries Used

The app is basically built using eleventy js (11ty) for frontend, MongoDb Realm as a backend and mediapipe and tensorflow js for machine learning.

- Eleventy Js
- Mongo Db
- Mongo Db Realm functions
- Mongo Db Realm HTTPS Endpoints
- Mongo Db Atlas Search
- Tensorflow
- Mediapipe

### How I built it
Collecting 20,000+ ASL words was a difficult effort, therefore I wanted a backend that was simple and quick to set up. So, I choose Mongo Db realm as I was using Mongo Db to store my data.

#### Creating 20,000 Documents

The first time I saw my data, I wondered if I could even store this data into MongoDB and how difficult it would be. However, MongoDB Compass makes importing JSON data extremely easy.

![Mongo Db Compass](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f7zszogbdegkw9circu0.png)

#### Mongo Db Atlas Search

Creating a Search Index for my data collection was a breeze, and I was impressed with how fast and precise the results were.


![atlas search](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jk4jwxq622jkar4jbwbu.png)

#### Mongo Db Realm HTTPS ENDPOINTS

After adding data and creating search index, I wanted to create an API that could be used by the frontend. Mongo DB makes creating api easy through their HTTPS ENDPOINTS & realm function services.

I built a search Index endpoint called `/searchVideo` which took a realm function and returns search results from atlas search.

```jsx

exports = function(request,response){
  let collection = context.services.get("mongodb-atlas").db("SignLanguage").collection("videos");
  const {searchVid} = request.query
  
    let pipeline = [
        {
          $search: {
            index: 'searchVideos',
            text: {
              query:searchVid,
              path: {
                'wildcard': '*'
              }
            }
          }
        }
    ];
    
    return collection.aggregate(pipeline)
};

```

I also built an simple API endpoint called `videoApi` and `alphabetSong` which returns paginated data from database.


```jsx
exports = function(request,response){
  const {start} = request.query
  
  let data = context.services.get("mongodb-atlas").db("SignLanguage").collection("videos").find().skip(parseInt(start)).limit(12).toArray();
  
  return data;
};
```


![realm function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ss85vzbro522avvgyi8i.png)



For creating ASL game, One of the issues I experienced was creating and gathering datasets for deep learning model. I had to personally collect the dataset because the datasets obtained on Kaggle either did not predict results correctly or were of poor quality.

Machine Learning workflow model is shown below

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ratn6j5ncjergnsgsyyw.png)

A brief overview of creation of classifying model is;

- We first predict key-points on hands using the Open CV Python package and the mediapipe hands model by google.
- Following this, a custom model is created using Tensorflow that classifies the keypoint.
- This model is then converted to tfjs model which can be used in browser to predict hand signs.

![https://media.giphy.com/media/xUPOqo6E1XvWXwlCyQ/giphy.gif](https://media.giphy.com/media/xUPOqo6E1XvWXwlCyQ/giphy.gif)

## Thanks!
Hope you liked my project! if you have any feedback please feel free to comment down below.