Hugging Face's logo

davanstrien
/
flair

medical
Model card Files Community
davanstrien HF staff commited on
Commit
0fceb3f
•
1 Parent(s): 4f69fdf

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +144 -0
README.md ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## FLAIR: A Foundation LAnguage Image model of the Retina
2
+
3
+ <img src="https://github.com/jusiro/FLAIR/raw/main/documents/flair.png" width = "1000" alt="" align=center /> <br/>
4
+
5
+ <b>A Foundation LAnguage Image model of the Retina (FLAIR):</b> <br/>
6
+ <b>Encoding expert knowledge in text supervision</b> <br/>
7
+ [Julio Silva-Rodriguez<sup>1</sup>](https://scholar.google.es/citations?user=1UMYgHMAAAAJ&hl),
8
+ [Hadi Chakor<sup>2</sup>](https://scholar.google.ca/citations?user=0Njg-cQAAAAJ&hl),
9
+ [Riadh Kobbi<sup>2</sup>](https://ca.linkedin.com/in/riadh-kobbi),
10
+ [Jose Dolz<sup>1</sup>](https://scholar.google.es/citations?user=yHQIFFMAAAAJ&hl),
11
+ [Ismail Ben Ayed<sup>1</sup>](https://scholar.google.es/citations?user=29vyUccAAAAJ&hl) <br/>
12
+ [<sup>1</sup>LIVIA - ETS Montreal](https://liviamtl.ca/), [<sup>2</sup>DIAGNOS Inc.](https://www.diagnos.com/)<br/>
13
+ | [Project](https://jusiro.github.io/projects/flair) | [Paper](https://arxiv.org/pdf/2308.07898.pdf) | [Code](https://github.com/jusiro/FLAIR) | [Tutorials](https://colab.research.google.com/drive/1LE50MQmsEQxMM-qvytXGeJ9WAu09w1MR?usp=sharing) |
14
+
15
+ **note** this is a duplicate of the model card at https://github.com/jusiro/FLAIR/blob/main/readme.md updated to support loading from the Hugging Face Hub.
16
+
17
+ ## Install FLAIR
18
+
19
+ * Install in your enviroment a compatible torch version with your GPU. For example:
20
+ ```
21
+ conda create -n flair_env python=3.8 -y
22
+ conda activate flair_env
23
+ conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
24
+ ```
25
+
26
+ * Install FLAIR library.
27
+ ```
28
+ pip install git+https://github.com/jusiro/FLAIR.git
29
+ ```
30
+
31
+ ## Usage
32
+
33
+ ```
34
+ from PIL import Image
35
+ import numpy as np
36
+
37
+ # Import FLAIR
38
+ from flair import FLAIRModel
39
+ rom huggingface_hub import hf_hub_download
40
+
41
+ # Set model
42
+ model_path = hf_hub_download(repo_id="davanstrien/flair", filename="flair_resnet.pth")
43
+ model = FLAIRModel.load_from_pretrained(model_path)
44
+
45
+ # Load image and set target categories
46
+ # (if the repo is not cloned, download the image and change the path!)
47
+
48
+ image = np.array(Image.open("./documents/sample_macular_hole.png"))
49
+ text = ["normal", "healthy", "macular edema", "diabetic retinopathy", "glaucoma", "macular hole",
50
+ "lesion", "lesion in the macula"]
51
+
52
+ # Forward FLAIR model to compute similarities
53
+ probs, logits = model(image, text)
54
+
55
+ print("Image-Text similarities:")
56
+ print(logits.round(3)) # [[-0.32 -2.782 3.164 4.388 5.919 6.639 6.579 10.478]]
57
+ print("Probabilities:")
58
+ print(probs.round(3)) # [[0. 0. 0.001 0.002 0.01 0.02 0.019 0.948]]
59
+ ```
60
+
61
+ ## **Note**: problems during automatic **pre-trained weights download**
62
+
63
+ If you encounter any issue while downloading the **pre-trained weights** (i.e. `from_checkpoint=True`), you can manually download the weights from the following links (see Table), unzip the file, and store them at: `./flair/modeling/flair_pretrained_weights/[ID].pth`.
64
+
65
+ | Backbone | ID | |
66
+ |-----------|:------------:|:---------------------------------------------------------------------------------------------:|
67
+ | ResNet-50 | flair_resnet | [LINK](https://drive.google.com/file/d/1l24_2IzwQdnaa034I0zcyDLs_zMujsbR/view?usp=drive_link) |
68
+
69
+ ## Pre-training and transferability
70
+
71
+ In the following, we present the scripts for model pre-training and transferability. To use them, we recommend cloning the whole repository.
72
+
73
+ ```
74
+ git clone https://github.com/jusiro/FLAIR.git
75
+ cd FLAIR
76
+ pip install -r requirements.txt
77
+ ```
78
+
79
+ ### 📦 Foundation model pre-training
80
+
81
+ * Define the relative paths for datasets and dataframes in `./local_data/constants.py`.
82
+
83
+ * Prepare the FUNDUS assembly dataset - check `./local_data/prepare_partitions.py` to prepare the dataframes.
84
+
85
+
86
+ | | | | | | |
87
+ |---------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
88
+ | [01_EYEPACS](https://www.kaggle.com/datasets/mariaherrerot/eyepacspreprocess) | [08_ODIR-5K](https://www.kaggle.com/datasets/andrewmvd/ocular-disease-recognition-odir5k) | [15_APTOS](https://www.kaggle.com/competitions/aptos2019-blindness-detection/data) | [22_HEI-MED](https://github.com/lgiancaUTH/HEI-MED) | [29_AIROGS](https://zenodo.org/record/5793241#.ZDi2vNLMJH5) | [36_ACRIMA](https://biomedical-engineering-online.biomedcentral.com/articles/10.1186/s12938-019-0649-y) |
89
+ | [02_MESIDOR](https://www.adcis.net/en/third-party/messidor2/) | [09_PAPILA](https://figshare.com/articles/dataset/PAPILA/14798004/1) | [16_FUND-OCT](https://data.mendeley.com/datasets/trghs22fpg/3) | [23_HRF](http://www5.cs.fau.de/research/data/fundus-images/) | [30_SUSTech-SYSU](https://figshare.com/articles/dataset/The_SUSTech-SYSU_dataset_for_automated_exudate_detection_and_diabetic_retinopathy_grading/12570770/1) | [37_DeepDRiD](https://github.com/deepdrdoc/DeepDRiD) |
90
+ | [03_IDRID](https://idrid.grand-challenge.org/Rules/) | [10_PARAGUAY](https://zenodo.org/record/4647952#.ZBT5xXbMJD9) | [17_DiaRetDB1](https://www.it.lut.fi/project/imageret/diaretdb1_v2_1/) | [24_ORIGA](https://pubmed.ncbi.nlm.nih.gov/21095735/) | [31_JICHI](https://figshare.com/articles/figure/Davis_Grading_of_One_and_Concatenated_Figures/4879853/1) | |
91
+ | [04_RFMid](https://ieee-dataport.org/documents/retinal-fundus-multi-disease-image-dataset-rfmid-20) | [11_STARE](https://cecas.clemson.edu/~ahoover/stare/) | [18_DRIONS-DB](http://www.ia.uned.es/~ejcarmona/DRIONS-DB.html) | [25_REFUGE](https://refuge.grand-challenge.org/) | [32_CHAKSU](https://figshare.com/articles/dataset/Ch_k_u_A_glaucoma_specific_fundus_image_database/20123135?file=38944805) | |
92
+ | [05_1000x39](https://www.nature.com/articles/s41467-021-25138-w#Sec16) | [12_ARIA](https://www.damianjjfarnell.com/?page_id=276) | [19_Drishti-GS1](http://cvit.iiit.ac.in/projects/mip/drishti-gs/mip-dataset2/Home.php) | [26_ROC](http://webeye.ophth.uiowa.edu/ROC/) | [33_DR1-2](https://figshare.com/articles/dataset/Advancing_Bag_of_Visual_Words_Representations_for_Lesion_Classification_in_Retinal_Images/953671?file=6502302) | |
93
+ | [06_DEN](https://github.com/Jhhuangkay/DeepOpht-Medical-Report-Generation-for-Retinal-Images-via-Deep-Models-and-Visual-Explanation) | [13_FIVES](https://figshare.com/articles/figure/FIVES_A_Fundus_Image_Dataset_for_AI-based_Vessel_Segmentation/19688169/1) | [20_E-ophta](https://www.adcis.net/en/third-party/e-ophtha/) | [27_BRSET](https://physionet.org/content/brazilian-ophthalmological/1.0.0/) | [34_Cataract](https://www.kaggle.com/datasets/jr2ngb/cataractdataset) | |
94
+ | [07_LAG](https://github.com/smilell/AG-CNN) | [14_AGAR300](https://ieee-dataport.org/open-access/diabetic-retinopathy-fundus-image-datasetagar300) | [21_G1020](https://arxiv.org/abs/2006.09158) | [28_OIA-DDR](https://github.com/nkicsl/DDR-dataset) | [35_ScarDat](https://github.com/li-xirong/fundus10k) | |
95
+
96
+
97
+ * Vision-Language Pre-training.
98
+
99
+ ```
100
+ python main_pretrain.py --augment_description True --balance True --epochs 15 --batch_size 128 --num_workers 6
101
+ ```
102
+
103
+ ### 📦 Transferability to downstream tasks/domains
104
+ * Define the relative paths for datasets and dataframes in `./local_data/constants.py`.
105
+ * Prepare the experiment setting for the target dataset - we used `./local_data/experiments.py` to store them.
106
+
107
+ ```
108
+ if experiment == "02_MESSIDOR":
109
+ setting = {"dataframe": PATH_DATAFRAME_TRANSFERABILITY_CLASSIFICATION + "02_MESSIDOR.csv",
110
+ "task": "classification",
111
+ "targets": {"no diabetic retinopathy": 0,
112
+ "mild diabetic retinopathy": 1,
113
+ "moderate diabetic retinopathy": 2,
114
+ "severe diabetic retinopathy": 3,
115
+ "proliferative diabetic retinopathy": 4}}
116
+ ```
117
+
118
+ * Zero-shot (no adaptation).
119
+
120
+ ```
121
+ python main_transferability.py --experiment 02_MESSIDOR --method zero_shot --load_weights True --domain_knowledge True --shots_train 0% --shots_test 100% --project_features True --norm_features True --folds 1
122
+ ```
123
+
124
+ * Linear Probing.
125
+
126
+ ```
127
+ python main_transferability.py --experiment 02_MESSIDOR --method lp --load_weights True --shots_train 80% --shots_test 20% --project_features False --norm_features False --folds 5
128
+ ```
129
+
130
+ # Citation
131
+
132
+ If you find this repository useful, please consider citing this paper:
133
+ ```
134
+ @article{FLAIR2023,
135
+ title={A Foundation LAnguage-Image model of the Retina (FLAIR): Encoding expert knowledge in text supervision},
136
+ author={Julio Silva-Rodriguez and Hadi Chakor and Riadh Kobbi and Jose Dolz and Ismail Ben Ayed},
137
+ journal={ArXiv Preprint},
138
+ year={2023}
139
+ }
140
+ ```
141
+
142
+ # License
143
+
144
+ - **Code and Model Weights** are released under [Apache 2.0 license](LICENSE)