raman07's picture
public push
b465db6 verified
metadata
library_name: diffusers
pipeline_tag: text-to-image

Model Details

Model Description

This model is fine-tuned from stable-diffusion-v1-5 on 110,000 image-text pairs from the MIMIC dataset.

Model Sources

Direct Use

This model can be directly used to generate realistic medical images from text prompts.

How to Get Started with the Model

import os
from safetensors.torch import load_file
from diffusers.pipelines import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(sd_folder_path, revision="fp16")
exp_path = os.path.join('unet', 'diffusion_pytorch_model.safetensors')
state_dict = load_file(exp_path)

# Load the adapted U-Net
pipe.unet.load_state_dict(state_dict, strict=False)
pipe.to('cuda:0')

# Generate images with text prompts

TEXT_PROMPT = "No acute cardiopulmonary abnormality."
GUIDANCE_SCALE = 4
INFERENCE_STEPS = 75

result_image = pipe(
        prompt=TEXT_PROMPT,
        height=224,
        width=224,
        guidance_scale=GUIDANCE_SCALE,
        num_inference_steps=INFERENCE_STEPS,
    )

result_pil_image = result_image["images"][0]

Training Details

Training Data

This model has been fine-tuned on 110K image-text pairs from the MIMIC dataset.

Training Procedure

The training procedure has been described in detail in Section 4.3 of this paper.

Metrics

This model has been evaluated using the Fréchet inception distance (FID) Score on MIMIC dataset.

Results

Fine-Tuning Strategy FID Score
Full FT 58.74
Attention 52.41
Bias 20.81
Norm 29.84
Bias+Norm+Attention 35.93
LoRA 439.65
SV-Diff 23.59
DiffFit 42.50

Environmental Impact

Using Parameter-Efficient Fine-Tuning potentially causes lesser harm to the environment since we fine-tune a significantly lesser number of parameters in a model. This results in much lesser computing and hardware requirements.

Citation

BibTeX:

@article{dutt2023parameter, title={Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity}, author={Dutt, Raman and Ericsson, Linus and Sanchez, Pedro and Tsaftaris, Sotirios A and Hospedales, Timothy}, journal={arXiv preprint arXiv:2305.08252}, year={2023} }

APA:
Dutt, R., Ericsson, L., Sanchez, P., Tsaftaris, S. A., & Hospedales, T. (2023). Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity. arXiv preprint arXiv:2305.08252.

Model Card Authors

Raman Dutt
Twitter
LinkedIn
Email