--- license: apache-2.0 tags: - medical --- ## FLAIR: A Foundation LAnguage Image model of the Retina
A Foundation LAnguage Image model of the Retina (FLAIR):
Encoding expert knowledge in text supervision
[Julio Silva-Rodriguez1](https://scholar.google.es/citations?user=1UMYgHMAAAAJ&hl), [Hadi Chakor2](https://scholar.google.ca/citations?user=0Njg-cQAAAAJ&hl), [Riadh Kobbi2](https://ca.linkedin.com/in/riadh-kobbi), [Jose Dolz1](https://scholar.google.es/citations?user=yHQIFFMAAAAJ&hl), [Ismail Ben Ayed1](https://scholar.google.es/citations?user=29vyUccAAAAJ&hl)
[1LIVIA - ETS Montreal](https://liviamtl.ca/), [2DIAGNOS Inc.](https://www.diagnos.com/)
| [Project](https://jusiro.github.io/projects/flair) | [Paper](https://arxiv.org/pdf/2308.07898.pdf) | [Code](https://github.com/jusiro/FLAIR) | [Tutorials](https://colab.research.google.com/drive/1LE50MQmsEQxMM-qvytXGeJ9WAu09w1MR?usp=sharing) | **note** this is a duplicate of the model card at https://github.com/jusiro/FLAIR/blob/main/readme.md updated to support loading from the Hugging Face Hub. ## Install FLAIR * Install in your enviroment a compatible torch version with your GPU. For example: ``` conda create -n flair_env python=3.8 -y conda activate flair_env conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge ``` * Install FLAIR library. ``` pip install git+https://github.com/jusiro/FLAIR.git ``` * Install the huggingface_hub library. ``` pip install huggingface_hub ``` ## Usage ``` from PIL import Image import numpy as np # Import FLAIR from flair import FLAIRModel rom huggingface_hub import hf_hub_download # Set model model_path = hf_hub_download(repo_id="davanstrien/flair", filename="flair_resnet.pth") model = FLAIRModel.load_from_pretrained(model_path) # Load image and set target categories # (if the repo is not cloned, download the image and change the path!) image = np.array(Image.open("./documents/sample_macular_hole.png")) text = ["normal", "healthy", "macular edema", "diabetic retinopathy", "glaucoma", "macular hole", "lesion", "lesion in the macula"] # Forward FLAIR model to compute similarities probs, logits = model(image, text) print("Image-Text similarities:") print(logits.round(3)) # [[-0.32 -2.782 3.164 4.388 5.919 6.639 6.579 10.478]] print("Probabilities:") print(probs.round(3)) # [[0. 0. 0.001 0.002 0.01 0.02 0.019 0.948]] ``` ## **Note**: problems during automatic **pre-trained weights download** If you encounter any issue while downloading the **pre-trained weights** (i.e. `from_checkpoint=True`), you can manually download the weights from the following links (see Table), unzip the file, and store them at: `./flair/modeling/flair_pretrained_weights/[ID].pth`. | Backbone | ID | | |-----------|:------------:|:---------------------------------------------------------------------------------------------:| | ResNet-50 | flair_resnet | [LINK](https://drive.google.com/file/d/1l24_2IzwQdnaa034I0zcyDLs_zMujsbR/view?usp=drive_link) | ## Pre-training and transferability In the following, we present the scripts for model pre-training and transferability. To use them, we recommend cloning the whole repository. ``` git clone https://github.com/jusiro/FLAIR.git cd FLAIR pip install -r requirements.txt ``` ### 📦 Foundation model pre-training * Define the relative paths for datasets and dataframes in `./local_data/constants.py`. * Prepare the FUNDUS assembly dataset - check `./local_data/prepare_partitions.py` to prepare the dataframes. | | | | | | | |---------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------| | [01_EYEPACS](https://www.kaggle.com/datasets/mariaherrerot/eyepacspreprocess) | [08_ODIR-5K](https://www.kaggle.com/datasets/andrewmvd/ocular-disease-recognition-odir5k) | [15_APTOS](https://www.kaggle.com/competitions/aptos2019-blindness-detection/data) | [22_HEI-MED](https://github.com/lgiancaUTH/HEI-MED) | [29_AIROGS](https://zenodo.org/record/5793241#.ZDi2vNLMJH5) | [36_ACRIMA](https://biomedical-engineering-online.biomedcentral.com/articles/10.1186/s12938-019-0649-y) | | [02_MESIDOR](https://www.adcis.net/en/third-party/messidor2/) | [09_PAPILA](https://figshare.com/articles/dataset/PAPILA/14798004/1) | [16_FUND-OCT](https://data.mendeley.com/datasets/trghs22fpg/3) | [23_HRF](http://www5.cs.fau.de/research/data/fundus-images/) | [30_SUSTech-SYSU](https://figshare.com/articles/dataset/The_SUSTech-SYSU_dataset_for_automated_exudate_detection_and_diabetic_retinopathy_grading/12570770/1) | [37_DeepDRiD](https://github.com/deepdrdoc/DeepDRiD) | | [03_IDRID](https://idrid.grand-challenge.org/Rules/) | [10_PARAGUAY](https://zenodo.org/record/4647952#.ZBT5xXbMJD9) | [17_DiaRetDB1](https://www.it.lut.fi/project/imageret/diaretdb1_v2_1/) | [24_ORIGA](https://pubmed.ncbi.nlm.nih.gov/21095735/) | [31_JICHI](https://figshare.com/articles/figure/Davis_Grading_of_One_and_Concatenated_Figures/4879853/1) | | | [04_RFMid](https://ieee-dataport.org/documents/retinal-fundus-multi-disease-image-dataset-rfmid-20) | [11_STARE](https://cecas.clemson.edu/~ahoover/stare/) | [18_DRIONS-DB](http://www.ia.uned.es/~ejcarmona/DRIONS-DB.html) | [25_REFUGE](https://refuge.grand-challenge.org/) | [32_CHAKSU](https://figshare.com/articles/dataset/Ch_k_u_A_glaucoma_specific_fundus_image_database/20123135?file=38944805) | | | [05_1000x39](https://www.nature.com/articles/s41467-021-25138-w#Sec16) | [12_ARIA](https://www.damianjjfarnell.com/?page_id=276) | [19_Drishti-GS1](http://cvit.iiit.ac.in/projects/mip/drishti-gs/mip-dataset2/Home.php) | [26_ROC](http://webeye.ophth.uiowa.edu/ROC/) | [33_DR1-2](https://figshare.com/articles/dataset/Advancing_Bag_of_Visual_Words_Representations_for_Lesion_Classification_in_Retinal_Images/953671?file=6502302) | | | [06_DEN](https://github.com/Jhhuangkay/DeepOpht-Medical-Report-Generation-for-Retinal-Images-via-Deep-Models-and-Visual-Explanation) | [13_FIVES](https://figshare.com/articles/figure/FIVES_A_Fundus_Image_Dataset_for_AI-based_Vessel_Segmentation/19688169/1) | [20_E-ophta](https://www.adcis.net/en/third-party/e-ophtha/) | [27_BRSET](https://physionet.org/content/brazilian-ophthalmological/1.0.0/) | [34_Cataract](https://www.kaggle.com/datasets/jr2ngb/cataractdataset) | | | [07_LAG](https://github.com/smilell/AG-CNN) | [14_AGAR300](https://ieee-dataport.org/open-access/diabetic-retinopathy-fundus-image-datasetagar300) | [21_G1020](https://arxiv.org/abs/2006.09158) | [28_OIA-DDR](https://github.com/nkicsl/DDR-dataset) | [35_ScarDat](https://github.com/li-xirong/fundus10k) | | * Vision-Language Pre-training. ``` python main_pretrain.py --augment_description True --balance True --epochs 15 --batch_size 128 --num_workers 6 ``` ### 📦 Transferability to downstream tasks/domains * Define the relative paths for datasets and dataframes in `./local_data/constants.py`. * Prepare the experiment setting for the target dataset - we used `./local_data/experiments.py` to store them. ``` if experiment == "02_MESSIDOR": setting = {"dataframe": PATH_DATAFRAME_TRANSFERABILITY_CLASSIFICATION + "02_MESSIDOR.csv", "task": "classification", "targets": {"no diabetic retinopathy": 0, "mild diabetic retinopathy": 1, "moderate diabetic retinopathy": 2, "severe diabetic retinopathy": 3, "proliferative diabetic retinopathy": 4}} ``` * Zero-shot (no adaptation). ``` python main_transferability.py --experiment 02_MESSIDOR --method zero_shot --load_weights True --domain_knowledge True --shots_train 0% --shots_test 100% --project_features True --norm_features True --folds 1 ``` * Linear Probing. ``` python main_transferability.py --experiment 02_MESSIDOR --method lp --load_weights True --shots_train 80% --shots_test 20% --project_features False --norm_features False --folds 5 ``` # Citation If you find this repository useful, please consider citing this paper: ``` @article{FLAIR2023, title={A Foundation LAnguage-Image model of the Retina (FLAIR): Encoding expert knowledge in text supervision}, author={Julio Silva-Rodriguez and Hadi Chakor and Riadh Kobbi and Jose Dolz and Ismail Ben Ayed}, journal={ArXiv Preprint}, year={2023} } ``` # License - **Code and Model Weights** are released under [Apache 2.0 license](LICENSE)