pubmed-ophtha/experiment-checkpoints

PubMed-Ophtha Experiment Checkpoints

Every CLIP checkpoint behind the figures and tables of "Scientific Domain Knowledge Improves Vision-Language Fundus Models" (Hallitschke et al., 2026). The paper compares five sources of textual supervision for ophthalmic vision-language models (simplistic templates, augmented templates, medical reports, domain-specific literature, and general biomedical literature) under matched initialization, architecture, and training schedule, across five vision encoder architectures and three random seeds. This repository holds the resulting weights so the reported numbers can be reproduced without retraining.

These are the checkpoints evaluated in the paper, trained on the standard PubMed-Ophtha split with three seeds per configuration and scored on all 110 tasks. The single models in PubMed-Ophtha CLIP Models are a separate release, trained on a larger PubMed-Ophtha split and not part of the paper's evaluation.

Contents: Resources · What's here · Layout · Usage · Training · Intended use · Licensing · Citation

Resources

Preprint	arXiv:2605.02720
Dataset	pubmed-ophtha/PubMed-Ophtha
Dataset pipeline	berenslab/pubmed-ophtha
PDF parser	berenslab/pmo-parser
CLIP experiments	berenslab/pmo-experiments
Figure-parsing models	pubmed-ophtha/detection-models
PubMed-Ophtha CLIP	PubMed-Ophtha CLIP Models
Paper checkpoints	This repository

What's here

90 checkpoints, spanning:

Axis	Values
Vision encoder	ResNet-50, ResNet-50x4, ViT-B/16, ViT-B/32, ViT-L/14
Training data source	simplistic templates (Kaggle EyePACS), augmented templates (FLAIR), domain-specific literature (PubMed-Ophtha)
PubMed-Ophtha variants	CFP, CFP-matched, decontaminated, each across all five architectures
Seeds	three per combination

Two of the paper's training data sources are not included. The general biomedical literature baseline uses BiomedCLIP as released, without further finetuning. The medical reports models will be added later.

Checkpoints are PyTorch .pt files in the format written by OpenCLIP, containing the state_dict only.

Layout

The repository preserves the directory structure that pmo-experiments writes during training, so a downloaded copy can be fed straight back into the evaluation pipeline:

clip_models_{seed}/
    pmo-experiments_{architecture}_openai_{dataset_key}_bs{batch_size}_s{seed}_{timestamp}/
        config.yaml
        open_clip/checkpoints/epoch_50.pt

For example:

clip_models_106080954/
    pmo-experiments_RN50_openai_pubmed_ophtha_bs2048_s106080954_20260521_160417/
        config.yaml
        open_clip/checkpoints/epoch_50.pt

The experiment folder name encodes architecture, initialization, dataset key, batch size, seed, and the training timestamp. Only the final epoch is included. config.yaml is required for loading, since it carries the architecture string and training precision, and it also documents the full training configuration of that run.

Usage

With pmo-experiments

The intended path. load_checkpoint reads config.yaml, rebuilds the model, and loads the weights; passing epoch=None picks the highest available epoch:

from huggingface_hub import snapshot_download
from pmo_experiments.util.model_interface import load_checkpoint

root = snapshot_download("pubmed-ophtha/experiment-checkpoints", local_dir="checkpoints")
folder = (
    f"{root}/clip_models_106080954/"
    "pmo-experiments_RN50_openai_pubmed_ophtha_bs2048_s106080954_20260521_160417"
)

model, preprocess, tokenizer = load_checkpoint(folder, epoch=None, device="cuda")

Because the folder layout is preserved, pmo_experiments eval run also discovers the downloaded clip_models_* folders directly, which is the shortest route to reproducing the paper's tables:

pmo_experiments eval run --eval-split val
pmo_experiments eval run --eval-split test
pmo_experiments eval aggregate

With OpenCLIP alone

If you would rather not install pmo-experiments, load the checkpoint by hand. The architecture string must match the one in the folder name and in config.yaml:

import open_clip
import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    "pubmed-ophtha/experiment-checkpoints",
    "clip_models_106080954/"
    "pmo-experiments_RN50_openai_pubmed_ophtha_bs2048_s106080954_20260521_160417/"
    "open_clip/checkpoints/epoch_50.pt",
)

model, _, preprocess = open_clip.create_model_and_transforms("RN50", pretrained=None)
tokenizer = open_clip.get_tokenizer("RN50")

checkpoint = torch.load(path, map_location="cpu")
state_dict = {k.replace("module.", ""): v for k, v in checkpoint["state_dict"].items()}
model.load_state_dict(state_dict)
model.eval()

Training

All models were initialized from OpenAI's CLIP weights and trained for 50 epochs with a batch size of 2048, AdamW, mixed precision, and a cosine learning rate schedule with a 5-epoch linear warmup, on a single NVIDIA A100 (40GB) via OpenCLIP. The maximum learning rate was set per training data source to avoid divergence; all other hyperparameters were held constant so that performance differences are attributable to the training data. The full configuration of each run, including its learning rate, is in the config.yaml next to the weights.

Intended use

A research resource for reproducing and extending the paper's comparison of training data sources, and for studying how textual supervision shapes ophthalmic vision-language representations. These are not clinical models, were not validated for diagnostic use, and should not be used for patient care.

Licensing

MIT, matching the project's code and the OpenAI CLIP weights the models were initialized from. The checkpoints contain no training data; the corpora they were finetuned on keep their own terms.

Citation

@misc{hallitschke2026scientific,
      title={Scientific Domain Knowledge Improves Vision-Language Fundus Models},
      author={Verena Jasmin Hallitschke and Carsten Eickhoff and Philipp Berens},
      year={2026},
      eprint={2605.02720},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.02720},
}