Cloudflare AI Challenge: Audio Interaction with AudioInsight
Tags: cloudflarechallenge, devchallenge, ai, webdev
Published: 2024-04-14 21:32:35

*This is a submission for the [Cloudflare AI Challenge](https://dev.to/devteam/join-us-for-the-cloudflare-ai-challenge-3000-in-prizes-5f99).*

## What I Built

I built AudioInsight, an app that processes audio, transcribes it, summarizes it, generates a title for the content, and allows users to ask questions about the related audio.

The chat and audio are stored remotely, so the user can access later and ask new questions or rewatch the audio.

To make this app, I used some products from the Cloudflare catalog, such as: Cloudflare D1, R2, Workers AI, Cloudflare Pages, and AI Models: Automatic Speech Recognition, Summarization, and ​​Text Generation. These will be explained in the [Journey](#journey) section.

## Demo
[Demo Link](https://audioinsight.gabrielsena.dev/)
[Original Cloudflare Pages Demo Link](https://audioinsight-4on.pages.dev/)

{% embed https://youtu.be/T4-UL_MsW7I %}

## My Code
{% embed https://github.com/gabrielsenadev/audioinsight %}

## Journey

<!-- Tell us about your process, the task types and models you used, what you learned, anything you are particularly proud of, what you hope to do next, etc. -->

Working with AI is a curious thing to me. I was thinking about developing something using AI, and this challenge is enough to motivate me to do this. So, one of the incredible AI features for me is the capability to transform voice into text. Therefore, I decided to follow this path.

After some time thinking about what to do, I decided to process the audio, generate the content of the audio, summarize the content, and allow the user to ask questions about the uploaded audio.

I also decided to explore more of the Cloudflare ecosystem. Thus, one of my personal requirements is the capability to store chat and audio remotely and provide a way for the user to go back later.

After defining my requirements and goals, I started learning about the AI, how it works, and how Workers AI works. In this process, I decided to use these AI models: **a audio to text** (whisper), **a content summarization** (bart-large-cnn) and **text generation to answer questions and generate chat title** (neural-chat-7b-v3-1-awq).

In the [Multiple Models and/or Triple Task Types](#multiple-models-andor-triple-task-types) section, I explain how I use these models and show the application flow, which explains how I combine these AI models to participate in the **Additional Prize Category**.

After developing the main idea, I began to understand how Cloudflare databases and Cloudflare R2 work. Then, I implemented the capability to store user's chats and audio.

## Multiple Models and/or Triple Task Types

To create this app, I utilized **three** different AI model types to generate its content.

- [whisper](https://developers.cloudflare.com/workers-ai/models/whisper/) is responsible for **converting audio to text**.
- [bart-large-cnn](https://developers.cloudflare.com/workers-ai/models/bart-large-cnn/) is tasked with **generating a summary of the related audio content**.
- [neural-chat-7b-v3-1-awq](https://developers.cloudflare.com/workers-ai/models/neural-chat-7b-v3-1-awq/) handles **generating the chat title** and **answering questions** about the related content.

When the user uploads an audio, start the chat creation process. Here, **I combine all three models** to generate a piece of content for each: audio transcription, summarization, and chat title.

When the user asks a question, I only use the text generation AI model to answer the user's question.

**Application Flow**
Follow the flow below to understand how it works.

![An image with two flows shows how this application combines Cloudflare solutions, including three different AI models, to develop this app](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rfvy3r6s8e1ioqii3k94.png)

In this flow, we understand how I use these different AI models and how I utilized Cloudflare storage solutions to develop this app.

**Final words**

Developing this entry helped me understand more about how the AI ecosystem works and how I could use the Cloudflare ecosystem to empower my ideas into products.

Looking ahead, I'm considering incorporating private chats and additional chat features to enhance user interaction with audio.

Thank you for this challenge!