Cloudflare AI Hackathon: MedicalAI - Revolutionizing Healthcare with AI
Tags: cloudflarechallenge, devchallenge, ai, webdev
Published: 2024-04-14 18:17:00

*This is a submission for the [Cloudflare AI Challenge](https://dev.to/devteam/join-us-for-the-cloudflare-ai-challenge-3000-in-prizes-5f99).*

## What We Built

<!-- Tell us what your app does! -->
In the fast-paced world of healthcare, access to comprehensive patient data is paramount for **informed decision-making**. However, with data scattered across various sources such as text, images, and audio recordings (conversation between doctor and patient), extracting meaningful insights can be a daunting task for **doctors and healthcare professionals**.

That's where our innovative solution comes in. We're excited to introduce our hackathon project aimed at **empowering doctors** with a powerful tool to streamline data aggregation and analysis: **_Medical AI_**.

This is a team submission.
Team Member: [SenthilBalaji](https://dev.to/senthilbalajiganesan)

## Demo

Demo App Link: {% cta https://cloudflare-hackathon.pages.dev/ %} AI Medical Advisor {% endcta %}

{% youtube OqChJeTpUt0 %}
<!-- Share a link to your deployed solution on Workers or Pages, and add a screenshot or video embed of your app here. -->

## Our Code

<!-- Show us the code! Share a public link to your repo and be sure to include a README file with installation instructions. We also encourage you to add a license for your code.  --> 

{% cta https://github.com/sbbeez/cloudflare-hackathon %} Client Repo {% endcta %}


{% cta https://github.com/sanjaysekaren/summarize-dataSources-ai-models %} Worker repo {% endcta %}
## Journey

<!-- Tell us about your process, the task types and models you used, what you learned, anything you are particularly proud of, what you hope to do next, etc. -->

We have created a solution which will extract text from other data sources like **text** (observations from previous consultation), **image** (pic of the disease/affected area), **audio** (Discussion between patient and doctor or similar). And create a summarized patient data which will help the doctors to get clear idea about the patient current condition and history. 

**Key Features**

- **Process any data**: This feature allows to provide from different data source, it makes user easy to feed the data.
- **Diagnose** : Generate a clear summary from all the available text extracted and provide a clear context and precise idea.
- **Ask me anything**: This feature not only extract the data, but also vectorise it for interacting with it later. So medical professionals can raise specific questions about a scenario.

**Services used from Cloudflare**: 
- Workers AI
- Workers & Pages 
- Vectorize 
- R2

**Models used:**
1. `@cf/qwen/qwen1.5-14b-chat-awq` | LLM - For analysing and diagnosing patient data.
2. `@cf/baai/bge-base-en-v1.5` | text-to-embedding - For text embedding.
3. `@cf/openai/whisper` | audio-to-text - For text extraction from audio.
4. `@cf/unum/uform-gen2-qwen-500m` | image-to-text - For text extraction from image.
5. `@cf/facebook/bart-large-cnn` | summarization - For text summarization

We've developed a versatile client page that can handle various data types like images, text, and audio. Using advanced Cloudflare AI models, we extract text from images and audio recordings from AI models. Simultaneously, we embed this extracted text for vectorization, which is crucial for running prompts against the large data. By merging all extracted data, we utilize the summarization AI model for getting a quick idea, on top of that we are using `**@cf/qwen/qwen1.5-14b-chat-awq**` AI model to generate a clear and concise diagnose, providing a comprehensive overview of the scattered data. Additionally, the vectorization process enables our "Ask Me Anything" feature, allowing users to instantly query specific information from the provided data.

**Challenges Overcome:**
- Not able to extract relevant data from Summarization model, gives inappropriate details - `@cf/facebook/bart-large-cnn`.
- Not able to extract text from image uploaded using `@cf/unum/uform-gen2-qwen-500m` model.
- An alternate model for summary we used `@cf/qwen/qwen1.5-14b-chat-awq` model, which has max-length of content for only 4096.
- Some response from summary model and LLM is returning response in chinese
⁠⁠- Upgrading to pro version for workers had payment issues



**Multiple Models and/or Triple Task Types**

This Project utilized totally **Five** Models as mentioned above. 
And also we covered, **Four** task:
1. Text Extraction
2. Text Summarization
3. Text Embedding
4. Vectorizaiton - Using cloudflare vectorize

So, we are qualified for both Multiple Models and Triple Task Types.

Edited On: (15/04/2024)
Due to security concerns, just reverted the domain name in the blog post (No code change involved).

Looking forward to develop more projects using Cloudflare AI.
<!-- Let us know if your project utilized multiple models per task and/or if your project used three or more task types. If so, you may qualify for our additional prize categories! If not, please remove this section. -->

<!-- Team Submissions: Please pick one member to publish the submission and credit teammates by listing their DEV usernames directly in the body of the post. -->

<!-- Don't forget to add a cover image or an additional tag to this post (if you want). -->

<!-- Thanks for participating! -->