MedOptm — medical VLM checkpoints
Trained weights for MedOptm, a compact medical Vision-Language Model: a frozen vision encoder feeds visual tokens through a TokenPacker projector into Llama-3.2-1B, adapted with LoRA. You give it a medical/surgical image and a question; it answers in natural language.
- 💻 Code (required to load these weights): https://github.com/Devpatel1012/MedOptm-VLM
- 🗂️ Training dataset: https://huggingface.co/datasets/devpatel1012/polished-vlm-dataset
⚠️ Research use only — not for clinical use. Answers are derived from biomedical figure captions, not written or reviewed by clinicians, and can be wrong. Do not use this to make or support medical decisions.
Two models in this repo
Both are 1B-class and trained on the same curated Surgical-VQA data — they differ only in the vision encoder, which is the whole point of the comparison.
| Model | Vision encoder | How the vision side is trained | Checkpoint prefix |
|---|---|---|---|
| MedOptm-CvT (flagship) | CvT microsoft/cvt-w24-384-22k |
last encoder stage is fully fine-tuned | phase1_alignment_*, phase2_vqa_lora_* |
| MedOptm-ViT (efficient) | CLIP-ViT-L/14-336 | encoder adapted with LoRA (base frozen) | clipvit_phase1_alignment_*, clipvit_phase2_vqa_lora_* |
Each model was trained in two phases:
- Phase 1 —
*_alignment_*: aligns the projector on image–caption pairs. - Phase 2 —
*_vqa_lora_*: instruction-tunes on VQA (adds LoRA to the LLM).
Which file to use: the Phase 2 *_vqa_lora_best.pt of whichever model you
want — it is the final, self-contained model (it already includes the aligned
projector from Phase 1). *_periodic.pt files are just resume snapshots; the
Phase 1 files are the intermediate alignment stage and are not needed for inference.
These checkpoints store only the trained parts — the projector, the LoRA adapters (and, for the CvT model, the fine-tuned top encoder stage). The frozen base weights (
meta-llama/Llama-3.2-1Band the vision encoder) are not included: they're downloaded separately when you build the model with the repo code.
Results
On the curated Surgical-VQA held-out test split. Higher is better; MedOptm models are 1B-class.
| Model | Params | ROUGE-L | BLEU | METEOR | BERTScore-F1 |
|---|---|---|---|---|---|
| MedOptm-CvT (Curated) 🥇 | 1.58 B | 0.289 | 0.114 | 0.324 | 0.877 |
| MedOptm-ViT (Curated) 🥈 | 1.58 B | 0.282 | 0.112 | 0.298 | 0.875 |
| LLaVA-Med Mistral | 7.57 B | 0.219 | 0.025 | 0.267 | 0.869 |
| SmolVLM Instruct | 2.25 B | 0.147 | 0.015 | 0.155 | 0.864 |
| LLaVA-1.5 (general) | 7.06 B | 0.139 | 0.009 | 0.167 | 0.854 |
Headline: the 1.58 B MedOptm-CvT model beats LLaVA-Med 7B (0.289 vs 0.219 ROUGE-L, non-overlapping 95% CIs) at roughly 1/5 the size.
Efficiency
| Model | Peak VRAM | GFLOPs / sample | Tokens/s | ROUGE-L / GFLOP |
|---|---|---|---|---|
| MedOptm-CvT (Curated) | 10.28 GB | 1977 | 57.9 | 0.000146 |
| MedOptm-ViT (Curated) | 6.54 GB | 619 | 68.7 | 0.000455 |
Trade-off: MedOptm-ViT gives up only 2.6% ROUGE-L for **1/3 the FLOPs**,
the lowest VRAM, and the highest throughput — i.e. it is ~3× more compute-efficient
per unit of quality. Pick CvT for best quality, ViT for best efficiency.
Energy/FLOP figures are estimated (constant-GPU-draw × wall-clock), not power-meter measured.
Why the CvT checkpoint is much larger than the ViT one
The CvT model fully fine-tunes its top vision-encoder stage (hundreds of
millions of full-precision weights), whereas the ViT model adapts its encoder
with tiny LoRA layers. On top of that, these .pt files also store the
AdamW optimizer state (two fp32 momentum buffers per trained parameter),
which is the bulk of the size for the CvT model. That's why, e.g.,
phase1_alignment_best.pt (CvT) is ~2 GB while clipvit_phase1_alignment_best.pt
(ViT) is ~177 MB. The optimizer state is only needed to resume training — it can
be stripped for inference to shrink the files substantially.
How to use
These are not standalone HF models; load them with the project code.
git clone https://github.com/Devpatel1012/MedOptm-VLM.git
cd MedOptm-VLM && pip install -e .
# download this checkpoints repo (e.g. with huggingface_hub) alongside it
import torch
from medoptm.models.vit_vlm import ViTCustomVLM # or: from medoptm.models.cvt_vlm import CustomVLM
model = ViTCustomVLM(
hf_token="<your_hf_token>",
vision_model_name="openai/clip-vit-large-patch14-336",
llm_model_name="meta-llama/Llama-3.2-1B",
image_size=336, scale_factor=2,
vision_lora_r=16, vision_lora_alpha=32, vision_lora_dropout=0.05,
)
model.setup_llm_lora(lora_r=16, lora_alpha=32, lora_dropout=0.05)
ckpt = torch.load("clipvit_phase2_vqa_lora_best.pt", map_location="cpu", weights_only=True)
model.load_trainable_parameters(ckpt["model_state"])
model.eval()
answer = model.generate(images=[img], text_prompts=["Where were the pedicle screws placed?"])
For the CvT model, use CustomVLM with vision_model_name="microsoft/cvt-w24-384-22k",
scale_factor=1, and lora_r=64, lora_alpha=256, lora_dropout=0.10 (the
cvt_curated preset). Match each checkpoint to the right class and config —
the exact per-model settings are in medoptm/config.py.
Training data
Trained on devpatel1012/polished-vlm-dataset
— image–question–answer triples derived from BIOMEDICA figure captions
(BIOMEDICA is built from PubMed Central Open Access). See that dataset's card for
provenance and licensing.
Limitations
- Not clinically validated. Research artifact only.
- Results are measured on this project's curated Surgical-VQA test split; numbers may not transfer to other data.
- Absolute metric values are modest (medical VQA is hard, references are short); the comparison is meaningful relative to the baselines on the same split.
- Inherits any biases/gaps of the underlying BIOMEDICA data (published figures skew toward illustrative cases).
Licensing
These weights are a derivative of meta-llama/Llama-3.2-1B, so the Llama 3.2
Community License applies and the model must carry a "Built with Llama"
notice. The vision encoders (CvT / CLIP) and the training data carry their own
terms. Confirm and set the license field above accordingly before relying on it
commercially.
Citation
@misc{medoptm2026,
title = {MedOptm: a compact medical vision-language model},
author = {Dev Patel},
year = {2026},
howpublished = {\url{https://github.com/Devpatel1012/MedOptm-VLM}}
}
Model tree for devpatel1012/MedOptm-Checkpoints
Base model
meta-llama/Llama-3.2-1B