VISTA-PATH

An interactive foundation model for pathology image segmentation and quantitative analysis in computational pathology.

• 📄 Paper: https://www.arxiv.org/abs/2601.16451
• 💻 Code: https://github.com/zhihuanglab/VISTA-PATH
• 🔬 Integrated in TissueLab

VISTA-PATH integrates visual context, textual class prompts and expert-guided interaction. Pre-trained on over 1.6 million samples, it generalizes across organs and tissue types, supports human-in-the-loop refinement, and enables clinically interpretable analysis through survival-associated morphological features.

Architecture

Component	Source	Role
Text encoder	vinid/plip (CLIP text tower)	Encodes the class prompt "an image of {class_name}" as a token sequence
Segmentation trunk	facebook/mask2former-swin-small-ade-semantic	Swin encoder + pixel decoder + masked-attention transformer decoder
Box prompt encoder	facebook/sam-vit-base (frozen prompt_encoder)	Encodes an optional bounding-box prompt as two corner tokens

Twenty object queries cross-attend to the concatenated [text tokens ; box tokens] sequence before the Mask2Former decoder, and a per-query binary head produces a foreground probability map for the prompted class. When no box is given, two learnable "no-box" tokens replace the SAM corner tokens, so the same checkpoint serves both the prompt-free and the box-prompted settings.

Inference settings that must match the checkpoint: input resolution 512, 20 queries.

Files

pytorch_model.bin     # released weights (~506 MB)
models/backbones.py   # CustomSegmentationModel
inference.py          # prompt-free / tissue-box whole-slide inference
inference_bbx.py      # box-prompted inference driven by a label mask
inference_utils.py    # model construction, checkpoint loading, sliding windows
utils.py              # visualization + segmentation metrics
idx_to_names/         # label index -> class name mappings
examples_BRCA/        # toy ROI image/mask pairs

Usage

Download the repository and run either entrypoint. The three backbones above are pulled from the Hub on first run.

from huggingface_hub import snapshot_download
local = snapshot_download("zhihuanglab/VISTA-PATH")

Prompt-free segmentation of named classes over a slide:

python3 inference.py \
  --infer_vis_dir ./results \
  --checkpoint_file pytorch_model.bin \
  --image_file /path/to/slide.svs \
  --class_names "Tumor" "Stroma" \
  --crop_size 2048 --overlap 256 \
  --bbx_random 1

Box-prompted segmentation, where a label mask supplies one box per class present in each window:

python3 inference_bbx.py \
  --infer_vis_dir ./results \
  --json_file ./idx_to_names/BRCA.json \
  --checkpoint_file pytorch_model.bin \
  --image_dir ./examples_BRCA/images \
  --mask_dir ./examples_BRCA/masks \
  --bbx_random 0

--bbx_random is the probability of dropping the box prompt: 1 runs prompt-free, 0 conditions every window on its box. Only a .jpg overview is written by default; add --save_mask for the full-resolution label mask and --save_prob for per-class probability maps.

Environment: Python 3.12, torch 2.4.0+cu118 with torchvision 0.19.0+cu118, transformers 4.46.1, openslide-python with openslide-bin. The cu118 build covers sm_50 through sm_90 (V100 up to H200); Blackwell cards (RTX 50 series, B200) need a cu128 build instead - torch 2.10.0+cu128 with torchvision 0.25.0+cu128. See environment.yml and the installation section of the README in the GitHub repo.

Citation

@misc{liang2026vistapath,
  title        = {VISTA-PATH: An interactive foundation model for pathology image segmentation and quantitative analysis in computational pathology},
  author       = {Liang, Peixian and Li, Songhao and Koga, Shunsuke and Li, Yutong and Alipour, Zahra and Tang, Yucheng and Xu, Daguang and Huang, Zhi},
  year         = {2026},
  eprint       = {2601.16451},
  archivePrefix= {arXiv},
  primaryClass = {cs.CV},
  doi          = {10.48550/arXiv.2601.16451},
}

License

Released under the Penn Academic Software License Agreement (see LICENSE): non-commercial academic research use only, with attribution. Commercial use, sale, or other monetization of the model or its derivatives requires prior approval.