robi913/medsiglip-retinal-oct-lora

Model Card for MedSigLIP-OCT: Dual-Prompt Cross-Attention Fusion

This model is a fine-tuned adaptation of google/medsiglip-448 tailored for Retinal Optical Coherence Tomography (OCT) analysis. It introduces a novel Gated Bidirectional Cross-Attention Fusion mechanism to overcome the standard 64-token limit in Vision-Language Models, allowing the processing of highly detailed, multi-part clinical reports.

Model Details

Model Description

Standard Vision-Language Models (VLMs) enforce a strict 64-token limit on input text prompts. In medical OCT diagnostics, a single report must describe both the anatomical layer structure (morphology) and pathological biomarkers (lesions), which routinely exceeds 180–250 tokens. Standard truncation discards critical clinical context.

To solve this, our framework splits generated clinical descriptions into two focused sub-prompts (prompt_a for structural profile and prompt_b for pathological profile). These sub-prompts are processed via a Gated Bidirectional Cross-Attention Fusion module that produces a unified text embedding ($t_{final}$) mapped onto the same latent sphere as the visual embedding ($v$).

• Developed by: Robert-Emanuel Ardelean (Technical University of Cluj-Napoca)
• Shared by: Robert-Emanuel Ardelean
• Model type: Vision-Language Model (Multi-Task)
• Language(s) (NLP): English
• License: Health AI Developer Foundations (Inherited)
• Finetuned from model: google/medsiglip-448

Model Sources

• Finetuned from model: google/medsiglip-448
• Paper: "Text Embedding Cross Fusion for Overcoming Token Limits in Vision Language Models" (Accepted for CSSC UTCN 2026).
• Paper: "Domain Adaptation of Vision-Language Models for Retinal OCT Analysis via Multi-Task Latent Alignment and Cross-Attention Fusion" (Accepted for ICCP 2026).

Uses

Direct Use

The model provides aligned visual and textual embeddings for retinal OCT scans, enabling:

• Cross-Modal Retrieval: Image-to-Text (I2T) and Text-to-Image (T2I) retrieval.
• Zero-Shot & Supervised Classification: Classifying scans into AMD, DME, DRUSEN, or NORMAL categories.
• Severity Estimation: Continuous severity scoring based on an abstracted AREDS formulation.

Downstream Use

The architecture is designed to function as an experimental clinical triage or second-opinion assistant. It can be integrated with explainability tools (e.g., EigenCAM via SVD on the ViT activations) and uncertainty quantification methods (e.g., Monte Carlo Dropout) to assist researchers in analyzing retinal pathologies.

Out-of-Scope Use

Clinical Disclaimer: The multi-label biomarker detection heads and severity estimation features included in this pipeline ARE NOT medically validated for autonomous clinical decision-making. These features are experimental research prototypes. Do not use directly for patient diagnosis without strict clinical validation.

Bias, Risks, and Limitations

• Severity Propagation Issue: Severity estimation is tightly coupled with the predicted disease class. A misclassification (e.g., a normal scan classified as AMD) will artificially inflate the predicted severity score.
• Overconfidence: Experimental evaluations using Monte Carlo Dropout reveal systematic overconfidence. The Expected Calibration Error (ECE) is high, meaning the raw confidence scores should not be trusted blindly without temperature scaling.
• Weak Labels: The biomarker training pipeline utilized secondary opinions and weak silver labels (YOLOv12 detections with a 0.25 confidence threshold) during the dataset expansion phase.

Recommendations

Users should be aware that the model's performance on highly heterogeneous classes like AMD (which spans from early drusen to advanced geographic atrophy) might fluctuate. Always use this model strictly as an assistive tool, keeping a human-in-the-loop.

How to Get Started with the Model

Use the code below to get started with the model.

Note: Since the standard Hugging Face AutoModel does not natively contain our custom cross-attention fusion layers, you must implement or import the custom wrapper classes (MedSigLIPMultiTask and CrossAttentionFusion) from the source code.

import torch
from PIL import Image
from transformers import AutoProcessor

# NOTE: Import your custom model class that implements the fusion logic
# from src.model.medsiglip import MedSigLIPMultiTask

model_id = "google/medsiglip-448"
processor = AutoProcessor.from_pretrained(model_id)

# 1. Load the custom model architecture and weights
# model = MedSigLIPMultiTask(base_model=model_id)
# checkpoint = torch.load("path_to_model/final_with_probe.pth")
# model.load_state_dict(checkpoint)
# model.eval()

# 2. Load OCT image
image = Image.open("sample_oct.jpg")

# 3. Dual Prompts Example (Split Semantic Texts)
prompt_structural = "Age-Related Macular Degeneration presents with an average retinal thickness of 85.1 pixels..."
prompt_pathological = "Age-Related Macular Degeneration manifests as four distinct lesions including geographic atrophy..."

# 4. Process inputs
# Visual features
inputs_img = processor(images=image, return_tensors="pt")

# Text features (truncated safely to 64 tokens each)
inputs_text_a = processor(text=prompt_structural, return_tensors="pt", padding=True, truncation=True, max_length=64)
inputs_text_b = processor(text=prompt_pathological, return_tensors="pt", padding=True, truncation=True, max_length=64)

# 5. Forward pass (pseudo-code depending on the custom wrapper implementation)
with torch.no_grad():
    # image_features = model.encode_image(inputs_img.pixel_values)
    # text_features = model.cross_attention_fusion(inputs_text_a, inputs_text_b)
    
    # Calculate Cosine Similarity or pass to classification heads...
    pass

Training Details

Training Data

The model was fine-tuned on the OCT5k dataset, comprising 4,596 images across 4 classes (AMD, DME, Drusen, Normal). Splits were rigorously constructed at the patient level (splits_v3) to prevent data leakage between training and testing sets.

Training Procedure

Preprocessing

• Biomarker Detection: YOLOv12e was used to generate bounding boxes on images lacking manual annotations.
• Prompt Generation: MedGemma 27B generated dense 180-256 token clinical reports based on the OCT images, layer segmentations, and bounding boxes.
• Prompt Splitting: Gemini Flash-Lite was utilized to semantically split the long descriptions into prompt_a (anatomy) and prompt_b (pathology), fitting the 64-token encoder limit.

Training Hyperparameters

• Training regime: Mixed precision multi-task learning.
• Fine-Tuning Strategy: LoRA applied to the visual encoder ($r=16, \alpha=32$).
• Multi-Task Loss Weights:
  • SigLIP Contrastive Loss: 2.0x
  • Disease Classification (Cross-Entropy): 0.3x
  • Severity Regression (Smooth L1): 0.2x
• Classification Head Optimization: Linear Probing was used post-training to optimize the classification head without degrading the learned contrastive retrieval space.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluated on the patient-disjoint test split (splits_v3) containing 748 OCT images.

Metrics

• Classification Accuracy
• F1 Macro
• Average Recall@1 (Retrieval)
• Severity Estimation (Mean Absolute Error - MAE)

Results

Metric	Baseline (CNN ResNet18 / Zero-Shot)	MedSigLIP Fine-Tuned (v15)
Classification Accuracy	66.8%	83.8%
F1 Macro	0.724	0.837
Average Recall@1	41.9%	84.8%
Severity MAE	—	23.3%

Summary

The fine-tuned MedSigLIP with Cross-Attention Fusion significantly outperforms the ResNet18 baseline and the Zero-Shot configuration across all clinical metrics. The model successfully aligns dense clinical context with microscopic OCT features.

Technical Specifications

Model Architecture and Objective

• Visual Encoder: ViT-based encoder from medsiglip-448, adapted via Low-Rank Adaptation (LoRA).
• Text Encoder: Frozen text encoder from the base model.
• Fusion Module: A custom Gated Bidirectional Cross-Attention module. It computes mutual attention between the structural and pathological embeddings, applies a learned sigmoidal gate, and uses a residual connection followed by L2 normalization to project $t_{final}$ into the shared space.
• Task Heads: Linear heads for 4-way classification, continuous severity regression, and 9 independent MLP heads for multi-label biomarker detection.

---

Acknowledgements: Developed at the Computer Science Department, Technical University of Cluj-Napoca (UTCN). Special thanks to the creators of the OCT5k dataset and Google for the MedSigLIP & MedGemma foundation models.