Hugging Face's logo

pykale
/
MeDSLIP

Zero-Shot Image Classification
English
medical
multimodal
vision-language pre-training
chest x-ray
Model card Files Community
MeDSLIP / README.md
wenruifan's picture
wenruifan
Update README.md
14181f3 verified
preview code
|
raw
history blame contribute delete
2.56 kB
metadata
license: cc-by-nc-4.0
language:
  - en
pipeline_tag: zero-shot-image-classification
tags:
  - medical
  - multimodal
  - vision-language pre-training
  - chest x-ray

MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment

Introduction:

The official implementation code for "MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment".

Arxiv Version

Quick Start:

Check checkpoints directory to download our pre-trained model from Hugging Face: MeDSLIP. It can be used for all zero-shot and finetuning tasks.

  • Zero-Shot Classification:

    We give an example on CXR14 in Sample_Zero-Shot_Classification_CXR14. Change the data paths, and test our model by python test.py. We give an example on RSNA in Sample_Zero-Shot_Classification_RSNA. Change the data paths, and test our model by python test.py.

  • Zero-Shot Grounding:

    We give an example on RSNA_Pneumonia in Sample_Zero-Shot_Grounding_RSNA. Change the data paths, and test our model by python test.py.

  • Finetuning:

    We give segmentation and classification finetune code on SIIM_ACR dataset in Sample_Finetuning_SIIMACR. Change the data paths, and finetune our model by python I1_classification/train_res_ft.py or python I2_segementation/train_res_ft.py.

Pre-train:

Data Preparation

All files for data preparation files can be downloaded from Hugging Face: MeDSLIP.

  • Extracted triplets: landmark_observation_adj_mtx.npy
  • Training list: train.json
  • Validation list: valid.json
  • Test list: test.json

Pre-training

Our pre-train code is given in PreTrain_MeDSLIP.

  • Check the PreTrain_MeDSLIP/data_file dir and download the files for data preparation.
  • Change the data and preparation files paths as you disire in PreTrain_MeDSLIP/configs/Pretrain_MeDSLIP.yaml, and python PreTrain_MeDSLIP/train_MeDSLIP.py to pre-train.

Reference 

@article{fan2024medslip,
  title={MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment},
  author={Fan, Wenrui and Suvon, Mohammod Naimul Islam and Zhou, Shuo and Liu, Xianyuan and Alabed, Samer and Osmani, Venet and Swift, Andrew and Chen, Chen and Lu, Haiping},
  journal={arXiv preprint arXiv:2403.10635},
  year={2024}
}

Contact

If you have any question, please feel free to contact winslow.fan@outlook.com.