MedVTok: A General-Purpose Medical Visual Tokenizer

GitHub · Project Page

MedVTok encodes 2D medical images and 3D volumes into a unified token space, preserving anatomical structure and clinical semantics across imaging modalities.

Highlights

• Broad coverage: trained on 28M+ images and 70K+ volumes across 9 medical imaging modalities.
• Reusable representations: provides global features, spatial patch tokens, and continuous latents for feature extraction and reconstruction.
• Versatile evaluation: evaluated on 24 benchmarks spanning classification, retrieval, segmentation, image synthesis, visual question answering, and report generation.

Download and usage

Follow the setup guide, then run the following from the MedVTok repository root:

hf download massaki75/medvtok config.json ckpt.pth --local-dir weights

import torch
from PIL import Image
from models.medvtok import build_model, build_transform

device = "cuda" if torch.cuda.is_available() else "cpu"
model = build_model(
    arch="medvtok-clip-meanpool",
    pretrained_path="weights/ckpt.pth",
    device=device,
).eval()

transform = build_transform(image_shape=(256, 256))
image = transform(Image.open("data/vis_imgs/sample00.png").convert("RGB"))

with torch.inference_mode():
    outputs = model(
        [image.to(device)],
        sample_posterior=False,
        decode=True,
        return_batched_tensor=False,
    )

features = outputs["feats"]                 # (1, 1152)
patch_tokens = outputs["patch_tokens"]      # (1, 256, 1152)
latents = outputs["latent"]                # (1, 256, 48)
reconstruction = outputs["recons"][0]      # (3, 256, 256)

Use decode=False for feature extraction without reconstruction. For 3D inputs, pass a list of normalized [3, D, H, W] tensors; see the 2D/3D demo for volume loading and preprocessing.

Citation

If you use MedVTok in your research, please cite:

@inproceedings{ma2026medvtok,
  title  = {MedVTok: A General-Purpose Medical Visual Tokenizer},
  author = {Ma, Chenglong and Ji, Yuanfeng and Ning, Junzhi and Liu, Jiyao and Wei, Yujie and Rui, Shaohao and Zhang, Chaoyang and Li, Wenjie and Tang, Wenhao and Tang, Cheng and Li, Wei and Gao, Shujian and Zhang, Yujie and Chen, Ying and Li, Zilong and Ge, Zongyuan and Song, Diping and Ye, Jin and Hu, Ming and He, Junjun and Shan, Hongming},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year   = {2026}
}