MedOptm — medical VLM checkpoints

Trained weights for MedOptm, a compact medical Vision-Language Model: a frozen vision encoder feeds visual tokens through a TokenPacker projector into Llama-3.2-1B, adapted with LoRA. You give it a medical/surgical image and a question; it answers in natural language.

⚠️ Research use only — not for clinical use. Answers are derived from biomedical figure captions, not written or reviewed by clinicians, and can be wrong. Do not use this to make or support medical decisions.


Two models in this repo

Both are 1B-class and trained on the same curated Surgical-VQA data — they differ only in the vision encoder, which is the whole point of the comparison.

Model Vision encoder How the vision side is trained Checkpoint prefix
MedOptm-CvT (flagship) CvT microsoft/cvt-w24-384-22k last encoder stage is fully fine-tuned phase1_alignment_*, phase2_vqa_lora_*
MedOptm-ViT (efficient) CLIP-ViT-L/14-336 encoder adapted with LoRA (base frozen) clipvit_phase1_alignment_*, clipvit_phase2_vqa_lora_*

Each model was trained in two phases:

  • Phase 1 — *_alignment_*: aligns the projector on image–caption pairs.
  • Phase 2 — *_vqa_lora_*: instruction-tunes on VQA (adds LoRA to the LLM).

Which file to use: the Phase 2 *_vqa_lora_best.pt of whichever model you want — it is the final, self-contained model (it already includes the aligned projector from Phase 1). *_periodic.pt files are just resume snapshots; the Phase 1 files are the intermediate alignment stage and are not needed for inference.

These checkpoints store only the trained parts — the projector, the LoRA adapters (and, for the CvT model, the fine-tuned top encoder stage). The frozen base weights (meta-llama/Llama-3.2-1B and the vision encoder) are not included: they're downloaded separately when you build the model with the repo code.


Results

On the curated Surgical-VQA held-out test split. Higher is better; MedOptm models are 1B-class.

Model Params ROUGE-L BLEU METEOR BERTScore-F1
MedOptm-CvT (Curated) 🥇 1.58 B 0.289 0.114 0.324 0.877
MedOptm-ViT (Curated) 🥈 1.58 B 0.282 0.112 0.298 0.875
LLaVA-Med Mistral 7.57 B 0.219 0.025 0.267 0.869
SmolVLM Instruct 2.25 B 0.147 0.015 0.155 0.864
LLaVA-1.5 (general) 7.06 B 0.139 0.009 0.167 0.854

Headline: the 1.58 B MedOptm-CvT model beats LLaVA-Med 7B (0.289 vs 0.219 ROUGE-L, non-overlapping 95% CIs) at roughly 1/5 the size.

Efficiency

Model Peak VRAM GFLOPs / sample Tokens/s ROUGE-L / GFLOP
MedOptm-CvT (Curated) 10.28 GB 1977 57.9 0.000146
MedOptm-ViT (Curated) 6.54 GB 619 68.7 0.000455

Trade-off: MedOptm-ViT gives up only 2.6% ROUGE-L for **1/3 the FLOPs**, the lowest VRAM, and the highest throughput — i.e. it is ~3× more compute-efficient per unit of quality. Pick CvT for best quality, ViT for best efficiency.

Energy/FLOP figures are estimated (constant-GPU-draw × wall-clock), not power-meter measured.


Why the CvT checkpoint is much larger than the ViT one

The CvT model fully fine-tunes its top vision-encoder stage (hundreds of millions of full-precision weights), whereas the ViT model adapts its encoder with tiny LoRA layers. On top of that, these .pt files also store the AdamW optimizer state (two fp32 momentum buffers per trained parameter), which is the bulk of the size for the CvT model. That's why, e.g., phase1_alignment_best.pt (CvT) is ~2 GB while clipvit_phase1_alignment_best.pt (ViT) is ~177 MB. The optimizer state is only needed to resume training — it can be stripped for inference to shrink the files substantially.


How to use

These are not standalone HF models; load them with the project code.

git clone https://github.com/Devpatel1012/MedOptm-VLM.git
cd MedOptm-VLM && pip install -e .
# download this checkpoints repo (e.g. with huggingface_hub) alongside it
import torch
from medoptm.models.vit_vlm import ViTCustomVLM   # or: from medoptm.models.cvt_vlm import CustomVLM

model = ViTCustomVLM(
    hf_token="<your_hf_token>",
    vision_model_name="openai/clip-vit-large-patch14-336",
    llm_model_name="meta-llama/Llama-3.2-1B",
    image_size=336, scale_factor=2,
    vision_lora_r=16, vision_lora_alpha=32, vision_lora_dropout=0.05,
)
model.setup_llm_lora(lora_r=16, lora_alpha=32, lora_dropout=0.05)

ckpt = torch.load("clipvit_phase2_vqa_lora_best.pt", map_location="cpu", weights_only=True)
model.load_trainable_parameters(ckpt["model_state"])
model.eval()

answer = model.generate(images=[img], text_prompts=["Where were the pedicle screws placed?"])

For the CvT model, use CustomVLM with vision_model_name="microsoft/cvt-w24-384-22k", scale_factor=1, and lora_r=64, lora_alpha=256, lora_dropout=0.10 (the cvt_curated preset). Match each checkpoint to the right class and config — the exact per-model settings are in medoptm/config.py.


Training data

Trained on devpatel1012/polished-vlm-dataset — image–question–answer triples derived from BIOMEDICA figure captions (BIOMEDICA is built from PubMed Central Open Access). See that dataset's card for provenance and licensing.

Limitations

  • Not clinically validated. Research artifact only.
  • Results are measured on this project's curated Surgical-VQA test split; numbers may not transfer to other data.
  • Absolute metric values are modest (medical VQA is hard, references are short); the comparison is meaningful relative to the baselines on the same split.
  • Inherits any biases/gaps of the underlying BIOMEDICA data (published figures skew toward illustrative cases).

Licensing

These weights are a derivative of meta-llama/Llama-3.2-1B, so the Llama 3.2 Community License applies and the model must carry a "Built with Llama" notice. The vision encoders (CvT / CLIP) and the training data carry their own terms. Confirm and set the license field above accordingly before relying on it commercially.

Citation

@misc{medoptm2026,
  title  = {MedOptm: a compact medical vision-language model},
  author = {Dev Patel},
  year   = {2026},
  howpublished = {\url{https://github.com/Devpatel1012/MedOptm-VLM}}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for devpatel1012/MedOptm-Checkpoints

Adapter
(724)
this model

Dataset used to train devpatel1012/MedOptm-Checkpoints