metadata

title: Medical_RAG
app_file: combinedmultimodal.py
sdk: gradio
sdk_version: 4.41.0

Advancing Text Searching with Advanced Indexing Techniques in Healthcare Applications(In Progress)

Welcome to the project repository for advancing text searching with advanced indexing techniques in healthcare applications. This project implements a powerful Retrieval-Augmented Generation (RAG) system using cutting-edge AI technologies, specifically designed to enhance text searching capabilities within the healthcare domain.I have also implemented Multimodal Text Searching for Medical Documents.

🚀 Features For Text Based Medical Query Based System

BioLLM 8B: Advanced language model for generating and processing medical text.
ClinicalBert: State-of-the-art embedding model for accurate representation of medical texts.
Qdrant: Self-hosted Vector Database (Vector DB) for efficient storage and retrieval of embeddings.
Langchain & Llama CPP: Orchestration frameworks for seamless integration and workflow management.

Medical Knowledge Base Query System

A multimodal medical information retrieval system combining text and image-based querying for comprehensive medical knowledge access.

Features For Multimodality Medical Query Based System:

Watch the video on YouTube

🧠 Multimodal Medical Information Retrieval

Combines text and image-based querying for comprehensive medical knowledge access
Uses Qdrant vector database to store and retrieve both text and image embeddings

🔤 Advanced Natural Language Processing

Utilizes ClinicalBERT for domain-specific text embeddings
Implements NVIDIA's Palmyra-med-70b model for medical language understanding fast Inference time.

🖼️ Image Analysis Capabilities

Incorporates CLIP (Contrastive Language-Image Pre-training) for image feature extraction
Generates image summaries using Google's Gemini 1.5 Flash model

📄 PDF Processing

Extracts text and images from medical PDF documents
Implements intelligent chunking strategies for text processing

🔍 Vector Search

Uses Qdrant for efficient similarity search on both text and image vectors
Implements hybrid search combining CLIP-based image similarity and text-based summary similarity

🖥️ Interactive User Interface

Gradio-based web interface for easy querying and result visualization
Displays relevant text responses alongside related medical images

🧩 Extensible Architecture

Modular design allowing for easy integration of new models or data sources
Supports both local and cloud-based model deployment The high level architectural framework for this application is given as follows:

⚡ Performance Optimization

Implements batching and multi-threading for efficient processing of large document sets
Utilizes GPU acceleration where available

🎛️ Customizable Retrieval

Adjustable similarity thresholds for image retrieval
Configurable number of top-k results for both text and image queries

📊 Comprehensive Visualization

Displays query results with both textual information and related images
Provides a gallery view of all extracted images from the knowledge base

🔐 Environment Management

Uses .env file for secure API key management
Supports both CPU and GPU environments

DEMO SCREENSHOT

🎥 Video Demonstration

Explore the capabilities of our project with our detailed YouTube video.

Installation

To get started with this project, follow these steps:

Install Dependencies:
```
pip install -r requirements.txt
```
Set up Qdrant:
- Follow the Qdrant Installation Guide to install and configure Qdrant.
Configure the Application:
- Ensure configuration files for BioLLM, ClinicalBert, Langchain, and Llama CPP are correctly set up.
Run the Application: if you want to run the text reterival application in Flask mode
```
uvicorn app:app
```

if you want to run the text reterival application through Streamlit ``bash streamlit run Streaming.py


if you want to run the multimodal application run it through Gradio Interface
```bash
python combinedmultimodal.py

💡 Usage

Querying the System: Input medical queries via the application's interface for detailed information retrieval.
Text Generation: Utilize BioLLM 8B to generate comprehensive medical responses.

👥 Contributing

We welcome contributions to enhance this project! Here's how you can contribute:

Fork the repository.
Create a new branch (git checkout -b feature-name).
Commit your changes (git commit -am 'Add feature').
Push to the branch (git push origin feature-name).
Open a Pull Request with detailed information about your changes.

📜 License

This project is licensed under the MIT License. See the LICENSE file for details.

📞 Contact

For questions or suggestions, please open an issue or contact the repository owner at surbhisharma9099@gmail.com.