library_name: diffusers
pipeline_tag: text-to-image
Model Details
Model Description
This model is fine-tuned from stable-diffusion-v1-5 on 110,000 image-text pairs from the MIMIC dataset using the attention-tuning PEFT method. Under this fine-tuning strategy, fine-tune only the attention weights in the U-Net while keeping everything else frozen. Attention tuning was first shown to be effective for fine-tuning vision transformers in this paper
- Developed by: Raman Dutt
- Shared by: Raman Dutt
- Model type: [Stable Diffusion fine-tuned using Parameter-Efficient Fine-Tuning]
- Finetuned from model: stable-diffusion-v1-5
Model Sources
- Paper: Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity
- Demo: MIMIC-SD-PEFT-Demo
Direct Use
This model can be directly used to generate realistic medical images from text prompts.
How to Get Started with the Model
from diffusers.pipelines import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained(sd_folder_path, revision="fp16")
Training Details
Training Data
[More Information Needed]
Training Procedure
Metrics
This model has been evaluated using the Fréchet inception distance (FID) Score on MIMIC dataset.
Results
[More Information Needed]
Environmental Impact
Citation
BibTeX:
@article{dutt2023parameter, title={Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity}, author={Dutt, Raman and Ericsson, Linus and Sanchez, Pedro and Tsaftaris, Sotirios A and Hospedales, Timothy}, journal={arXiv preprint arXiv:2305.08252}, year={2023} }
APA:
Dutt, R., Ericsson, L., Sanchez, P., Tsaftaris, S. A., & Hospedales, T. (2023). Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity. arXiv preprint arXiv:2305.08252.