pamessina/medgemma-4b-it-cure

CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation

Accepted as an Oral Presentation at CVPR 2026.

CURE (Curriculum-guided Multi-task Training) is an error-aware curriculum learning framework that improves grounding and report quality for medical vision-language models without any additional data.

Grounded medical Vision-Language Models often learn to associate the task of visual grounding exclusively with predicting abnormalities due to abnormality-biased datasets. This causes strong prior models to hallucinate abnormalities when asked to ground normal anatomy. CURE directly addresses this through a novel task structure and an adaptive training strategy.

Contributions & Takeaways

1. Anatomy-Grounded Report Generation (AGRG): A novel task that teaches the model to localize and describe specific anatomical regions. By decomposing reports (via Chest ImaGenome), CURE provides grounding supervision for both normal and abnormal anatomy, effectively mitigating abnormality bias.
2. Error-aware Curriculum: A dynamic adaptive sampling strategy designed to address dataset and class imbalance. Using a hybrid score (IoU + CXRFEScore), the curriculum prioritizes datasets and classes where validation performance is currently worse, focusing training on the hardest cases.

Main Takeaways: Ground and describe normal anatomy too, and use adaptive sampling to overcome medical data imbalance.

Key Results

• 67% relative reduction in abnormality hallucinations: Measured on average across six anatomical locations (e.g., clavicle hallucination rates dropped from ~60% down to just 1%).
• Better visual grounding: CURE consistently achieves higher IoU scores than baselines (like MAIRA-2) on Phrase Grounding (MS-CXR, PadChest-GR, VinDr-CXR) and Grounded Report Generation.
• Competitive report generation quality: By combining AGRG outputs over 29 distinct anatomical locations with standard GRG, CURE achieves report generation quality on MIMIC-CXR that matches or exceeds major baselines.

Model Details

• Developed by: Pablo Messina, Andrés Villa, Juan León Alcázar, Karen Sánchez, Carlos Hinojosa, Denis Parra, Álvaro Soto, Bernard Ghanem
• Affiliations: Pontificia Universidad Católica de Chile, King Abdullah University of Science and Technology (KAUST), CENIA, iHEALTH.
• Model type: Medical Vision-Language Model (VLM)
• Finetuned from model: google/medgemma-4b-it

How to Get Started with the Model

1. Try it in Google Colab

For the easiest way to run inference, including our custom image transformation pipeline (which is crucial for replicating optimal performance) and bounding box visualizations, please use our official Colab notebook:

2. Loading the Model Locally

You can load the base model and the CURE PEFT adapter using 4-bit quantization for efficient inference.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig
from peft import PeftModel

# Configuration
BASE_MODEL_ID = "google/medgemma-4b-it"
ADAPTER_ID = "pamessina/medgemma-4b-it-cure"

# 4-bit Quantization Config (Must match training settings)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_storage=torch.bfloat16,
)

# Load Model
base_model = AutoModelForImageTextToText.from_pretrained(
    BASE_MODEL_ID,
    quantization_config=quantization_config,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load the LoRA adapter onto the base model
model = PeftModel.from_pretrained(base_model, ADAPTER_ID)

# Load the processor
processor = AutoProcessor.from_pretrained(BASE_MODEL_ID)
processor.tokenizer.padding_side = "left"

model.eval()
print("CURE model and processor are ready for inference.")

Note: For the full inference script, including the required get_pil_with_augmentations_transforms Albumentations setup, please refer to the GitHub Repository or the Colab link above.

Citation

If you find this work useful, please cite our CVPR 2026 paper:

@InProceedings{Messina_2026_CVPR,
    author    = {Messina, Pablo and Villa, Andr\'es and Alcazar, Juan Leon and Sanchez, Karen and Hinojosa, Carlos and Parra, Denis and Soto, Alvaro and Ghanem, Bernard},
    title     = {CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {36279-36289}
}