babel_fish - real time language translation
Tags: cloudflarechallenge, devchallenge, ai, svelte
Published: 2024-04-14 21:47:15

*This is a submission for the [Cloudflare AI Challenge](https://dev.to/devteam/join-us-for-the-cloudflare-ai-challenge-3000-in-prizes-5f99).*

## What we built

Douglas Adams came up with the idea of the [Babel fish](https://hitchhikers.fandom.com/wiki/Babel_Fish) in Hitchhiker's Guide to the Galaxy. It's a fish that, once you've stuck it in your ear, allows any language to be translated to you in real time.

[`babel_fish`](https://trybabelfish.com) is a real time language translation app built with Cloudflare, Svelte, and multiple language models. The app allows you to share a room with your friends in different languages. All messages get translated to your chosen language in real time!

@alicelovescake and I thought this would be a perfect app for trying out the fancy new Cloudflare Workers AI!

## Demo

<!-- Share a link to your deployed solution on Workers or Pages, and add a screenshot or video embed of your app here. -->

{% embed https://www.loom.com/share/475b44db5d1647fea4ad9d55d54d0015?sid=113c3131-c1c9-4824-8a54-efbbe3eaa6f0 %}

## Le code

<!-- Show us the code! Share a public link to your repo and be sure to include a README file with installation instructions. We also encourage you to add a license for your code.  -->

### Frontend

The Svelte application contains the frontend code as well as all of our API endpoints interacting with the language models.

{% embed https://github.com/amorriscode/babel-fish %}


### Backend

Our web socket backend uses Cloudflare Workers and Durable Storage. It's a simple logic engine that powers the chat rooms.

{% embed https://github.com/amorriscode/babel-fish-ws %} 

## Journey

<!-- Tell us about your process, the task types and models you used, what you learned, anything you are particularly proud of, what you hope to do next, etc. -->

> Tired: trying out one new technology during a hackathon
> Wired: trying out tons of new technology during a hackathon

I've been admiring Cloudflare from afar for a while now. I love what they've been doing with Pages, Workers, and now AI Workers. This hackathon finally gave me a good excuse to play around with the platform!

Aside from all the fancy tooling Cloudflare provide, this was also my first time using Svelte (sorry Rich!). I actually really enjoyed the experience. I'd love to spend more time writing Svelte; I have a lot left to learn.

### So... AI?

We used a ton of Cloudflare's available AI models to build [`babel_fish`](https://trybabelfish.com).

- `@cf/openai/whisper` for voice-to-text
- `@hf/thebloke/mistral-7b-instruct-v0.1-awq` for language detection
- `@cf/meta/m2m100-1.2b` for language translation

Since a true Babel fish doesn't produce text we also used [ElevenLabs](https://elevenlabs.io/) for text-to-speech.

### Challenges

We only had a single day to work on our project. The tight timeline and all new tech made it an exciting 24 hours.

The biggest challenge of this project actually ended up being the [backend audio processing](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Using_AudioWorklet). With the initial POC, user messages weren't transcribed and sent over the wire until they were completely finished speaking. We were able to improve this by using [AudioWorklet](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet)s. Now we regularly check for gaps in a user's message and periodically process and broadcast messages.

The rest was easy because Cloudflare and Svelte made it so. 😎

Thanks to @cloudflaredev and @thepracticaldev for putting on this hackathon. If you're interested in this kind of thing, [come work on experimental AI projects with me at Stripe](https://stripe.com/jobs/listing/staff-engineer-developer-ai/5364470)!

<><