|
--- |
|
license: mit |
|
tags: |
|
- medical |
|
--- |
|
# MedImageInsight: Open-Source Medical Image Embedding Model |
|
|
|
This repository provides a simplified implementation for using the MedImageInsight model, an open-source medical imaging embedding model presented in the paper [MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging](https://arxiv.org/abs/2410.06542) by Noel C. F. Codella et al. The official guide to access the model from Microsoft is quite complicated, and it is arguable whether the model is truly open-source. This repository aims to make it easier to use the MedImageInsight model for various tasks, such as zero-shot classification, image embedding, and text embedding. |
|
|
|
What we have done: |
|
|
|
- Downloaded the models from Azure |
|
- Got rid of all the unnecessary files |
|
- Got rid of unnecessary MLflow code to make a standalone implementation |
|
- Moved to uv for dependency management |
|
- Added multi-label classification |
|
- Created an example with the FastAPI service |
|
|
|
## Usage |
|
|
|
1. Clone the repository and navigate to the project directory. |
|
|
|
Make sure you have git-lfs installed (https://git-lfs.com) |
|
```bash |
|
git lfs install |
|
``` |
|
|
|
```bash |
|
git clone https://huggingface.co/lion-ai/MedImageInsights |
|
``` |
|
3. Install the required dependencies |
|
We are using [uv](https://github.com/astral-sh/uv) package manager to simplify the installation. |
|
|
|
To create a virtual env, simply run: |
|
```bash |
|
uv sync |
|
``` |
|
Or to run a single script, just run: |
|
```bash |
|
uv run example.py |
|
``` |
|
|
|
That's it! |
|
|
|
## Examples |
|
See to the `example.py` file. |
|
### Zero-shot image classification |
|
|
|
Here's an example of how to use the `MedImageInsight` class for zero-shot classification: |
|
|
|
```python |
|
# Initialize classifier |
|
classifier = MedImageInsight( |
|
model_dir="2024.09.27", |
|
vision_model_name="medimageinsigt-v1.0.0.pt", |
|
language_model_name="language_model.pth" |
|
) |
|
|
|
# Load model |
|
classifier.load_model() |
|
|
|
# Read image |
|
image = base64.encodebytes(read_image("image.png")).decode("utf-8") |
|
|
|
# Zero-shot classification |
|
images = [image] |
|
labels = ["normal", "Pneumonia", "unclear"] |
|
results = classifier.predict(images, labels) |
|
print(results) |
|
``` |
|
### Multi-label zero-shot image classification |
|
Run multi-label image classification (without softmax at the end) |
|
```python |
|
# Multilabel classification example |
|
images = [image] |
|
labels = ["normal", "Pneumonia", "Fracture", "Tumor"] |
|
results = classifier.predict(images, labels, multilabel=True) |
|
print(results) |
|
|
|
``` |
|
### Image embeddings |
|
|
|
```python |
|
results = classifier.encode(images=images) |
|
print(results["image_embeddings"]) |
|
``` |
|
|
|
### Text embeddings |
|
```python |
|
results = classifier.encode(texts=labels) |
|
print(results["text_embeddings"]) |
|
``` |
|
### FastAPI server |
|
```bash |
|
uv run fastapi_app.py |
|
``` |
|
Go to localhost:8000/docs to see the swagger. |
|
|
|
The application provides endpoints for classification and image embeddings. Images have to be base64 encoded. |
|
|
|
|
|
## Roadmap |
|
- [x] Basic implementation |
|
- [x] Multilabel classification |
|
- [x] FastAPI service |
|
- [ ] HF compatible API (from_pretrained()) |
|
- [ ] Explainability |
|
|
|
## Acknowledgments |
|
|
|
This repository is based on the work presented in the paper "MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging" by Noel C. F. Codella et al. ([arXiv:2410.06542](https://arxiv.org/abs/2410.06542)). |
