Anime (or any other video) Translator - Chrome extension that automatically translates web videos! 👯
Tags: hackwithdg, javascript, deepgram, showdev
Published: 2022-04-11 23:24:09

## Overview of My Submission
![Anime OAV Screenshot](https://i.imgur.com/tTBj0Rr.png)
Anime OAV (Anime or any other video) is a chrome extension that automatically translate, transcribes, and generates overlaying subtitles for videos.

{% github gregorygaines/anime-aov-translator %}

## Tech Stack
- TailwindCSS - Ui style
- Chrome extension
- React.js - Front-end library
- Deepgram - Transcriber
- Azure / Google - Translator
- Webpack - Code bundler

### Submission Category: 
#### Accessibility Advocates


### Why
Sometimes my favorite or interesting shows are produced outside my native language and my only options are to wait for an official or fan-made translation.

I wondered what if I had a way to automatically translate videos outside my native language so I can enjoy them.

Translation isn't the only problem I've had. Sometimes audio can be hard to understand, and I think, if only I had a way to transcribe what everyone is saying.

My extension solves both of the issues above and increases awareness of issues that come with accessibility. 

### How does it work

#### Terminology
Below is a short description on how a chrome extension works.

![Chrome extension diagram](https://res.cloudinary.com/practicaldev/image/fetch/s--QhXYM9Xz--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://developer.chrome.com/static/images/overview/messagingarc.png)
Chrome extension diagram by [Aryclenio Barros](https://dev.to/aryclenio/how-to-create-a-google-chrome-extension-2d1g)

- [Content Scripts](https://developer.chrome.com/docs/extensions/mv2/content_scripts/) - Code that runs on the page the user is currently viewing. Commonly used for operating on the DOM and interacting with the current page. Because of security concerns, some actions can't be preformed on content scripts and must be done in a background script.

- [Background Scripts](https://developer.chrome.com/docs/extensions/mv2/service_workers/) - Code that runs in a service worker in the background and can't interact with the current page or DOM. Commonly used for long lived tasks or managing state across the extension. It can be thought of as a back-end of sorts for the extension. To communicate, the background and content scripts can register listeners to pass messages between each other. Since background scripts can preform actions content scripts can't, certain actions are executed there then passed to the content script.

### Translating / Transcribing
The background script captures the current tabs audio then passes the audio to **Deepgram** for transcription.

```javascript
const transcriber = 
SpeechRecognizerFactory
.createSpeechRecognizer(SpeechRecognizers.Deepgram);
```

After transcribing the audio, the extension uses one of its multiple translation engines to translate the text. For now, only **Azure** and **Google** are implemented.

```javascript
const translator = 
TranslatorFactory.createTranslator(Translators.AZURE);
...
const translator = 
TranslatorFactory.createTranslator(Translators.GOOGLE);
```

The translated text is then sent to the client as a message.

```javascript
const clientMessage = {
  command: "draw_text",
  data: translated,
}

chrome.tabs.sendMessage(activeTabId, clientMessage, (response) => {
  console.log("Sending text to client: " + translated);
});
```

#### Subtitles
Because of needed configurations, certain domains are white-listed for translations / transcriptions. When the client encounter a white-listed domain, the content script pulls the needed configuration to find the video player container and its dimensions and position, then draws a canvas overlay for displaying subtitles.

```javascript
const videoContainerSelector = {
  crunchyroll: "#showmedia_video_box",
}

const videoContainer = queryElement(crunchyroll);
const containerPos = getPos(videoContainer);
const containerDim = getDim(videoContainer);

renderSubtitleCanvas(containerPos, containerDim);
```

The content script then listens for subtitles from the background script so it can render them on the canvas.

```javascript
chrome.runtime.onMessage.addListener(
  (request, sender, sendResponse) => {
    switch (request.command) {
      case 'draw_text':
        console.log("Received command to draw text: " + request.data);
        drawTextToCanvas(request.data);
        sendResponse(true);
        return true;

      default:
        console.log("Unknown client command: " + request.command);
    }

    return true;
  },
);
```

### Features
- Audio transcribing
- Text translation 
- Render subtitles onto videos
- Multiple text translation engines: **Azure Cognitive Translator**, **Google Cloud Translation**

### Roadmap
- Ability to change complex options without modifying code
- Increase transcribing accuracy
- Add more transcribing engines
- Ability to customize subtitle font
- Clean up code!!

### Screenshots
<img width="100%" style="width:100%" src="https://media.giphy.com/media/DBCzvE37hDRvg6sN3b/giphy.gif">

### Conclusion
Not too bad for three days of work huh? This was an incredible project to work on. Sadly, I learned about the hackathon too late, but I had fun regardless.  

I want to continue updating the extension if there is a community interest to keep it alive. Open source contributions are extremely encouraged. Thanks for reading!