dmusingu/lapvqa-pretrain-contrastive

LAPVQA — Pretrain (Contrastive)

Part of the LAPVQA collection.

Description

A ViT-L/14 vision encoder trained from scratch on MIMIC-CXR chest X-ray / report pairs using InfoNCE contrastive learning (image encoder vs. 6-layer bidirectional text encoder). The encoder is intended to be used as a frozen feature extractor for downstream CXR tasks.

Architecture

Component	Detail
Vision backbone	ViT-L/14, 24-layer, 1024-dim, 16-head, patch 14, 384 px
Text encoder	6-layer, 512-dim bidirectional transformer, GPT-2 vocab (50 257)
Projection	Linear → 512-dim shared embedding space
Loss	InfoNCE (symmetric softmax cross-entropy)
Training data	MIMIC-CXR (physionet.org/content/mimic-cxr)
Epochs	50

Downstream Evaluation (frozen encoder + linear probe)

Dataset	Mean AUC
NIH CXR-14 (14-class)	0.653
CheXpert-5 (5-class)	0.759

Files

File	Description
encoder_final.pt	Vision encoder weights at the end of training
model_best.pt	Full model (encoder + text encoder) at best val loss

Usage

import torch
from lapvqa.pretrain.model import ContrastiveModel

ckpt = torch.load("encoder_final.pt", map_location="cpu")
model = ContrastiveModel()
model.vision_encoder.load_state_dict(ckpt)
model.eval()

Citation

If you use these weights please cite MIMIC-CXR:

@article{johnson2019mimic,
  title   = {MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports},
  author  = {Johnson, Alistair EW and others},
  journal = {Scientific data},
  volume  = {6}, pages = {317}, year = {2019}
}