You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
By accessing this model, you agree to use it only for research purposes and in accordance with the Apache 2.0 license. This model is intended for research use only and is not for clinical or diagnostic use.
Log in or Sign Up to review the conditions and access this model content.
Prov-GigaPath-Flash
A lightweight whole-slide foundation model for digital pathology
[Prov-GigaPath Model] [Code] [Paper] [BibTeX]
Prov-GigaPath-Flash is a lightweight variant of Prov-GigaPath, designed for faster and more memory-efficient tile- and slide-level inference and finetuning while retaining the same two-stage GigaPath design.
Model Overview
Overview of the GigaPath model architecture. Prov-GigaPath-Flash follows the same tile-encoder + slide-encoder design with lighter backbones.
Like Prov-GigaPath, the model consists of a tile encoder, which extracts local patterns at the patch (tile) level, and a slide encoder, which aggregates tile embeddings into a slide-level representation.
| Component | Architecture | Embedding dim | Params |
|---|---|---|---|
| Tile encoder | DINOv2-small ViT-S/16 (SwiGLU, LayerScale) | 384 | ~22M |
| Slide encoder | LongNet, 12 layers | 384 | ~21M |
Compared to Prov-GigaPath (ViT-giant tile encoder + 12L/768d slide encoder over 1536-dim tile embeddings), both the tile-embedding dimension and the slide encoder hidden dimension are 384.
Repository Contents
Following the original Prov-GigaPath packaging:
config.json+pytorch_model.bin— tile encoder (timm format)slide_encoder.pth— slide encoder ({"model": ...}, encoder weights only)
Install
Loading requires the Prov-GigaPath code,
which registers the GigaPath-Flash architectures (gigapath_tile_enc_dinov2s and
gigapath_slide_enc12l384d).
git clone https://github.com/prov-gigapath/prov-gigapath
cd prov-gigapath
conda env create -f environment.yaml
conda activate gigapath
pip install -e .
Model Download
Set your HuggingFace read-only token as an environment variable to access the weights:
export HF_TOKEN=<huggingface read-only token>
Inference
The model consists of a tile encoder, that extracts local patterns at patch level, and a slide encoder, that outputs representations at slide level. When doing inference at the slide level, we recommend following this pipeline: (1) Tile the whole slide into N image tiles, with the coordinates of each tile. (2) Get the embeddings for each tile using the tile encoder. (3) Pass the N image tile embeddings and their coordinates into the slide encoder, to get slide-level representations.
Inference with the tile encoder
from PIL import Image
from torchvision import transforms
import torch
import gigapath.tile_encoder as tile_encoder # registers gigapath_tile_enc_dinov2s
tile_enc = tile_encoder.create_model("hf_hub:prov-gigapath/prov-gigapath-flash")
transform = transforms.Compose(
[
transforms.Resize(256, interpolation=transforms.InterpolationMode.BICUBIC),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
]
)
sample_input = transform(Image.open("images/prov_normal_000_1.png").convert("RGB")).unsqueeze(0)
tile_enc.eval()
with torch.no_grad():
output = tile_enc(sample_input).squeeze() # 384-dim tile embedding
Inference with the slide encoder
To run inference with the slide encoder, provide both the tile embeddings and their coordinates:
import gigapath.slide_encoder as slide_encoder
slide_enc = slide_encoder.create_model(
"hf_hub:prov-gigapath/prov-gigapath-flash", "gigapath_slide_enc12l384d", 384
)
slide_enc.eval()
with torch.no_grad():
output = slide_enc(tile_embed, coordinates).squeeze() # slide-level embedding
Note Older versions of timm have compatibility issues. Please ensure you use a
newer version: pip install timm>=1.0.3.
Model Uses
Intended Use
The data, code, and model checkpoints are intended to be used solely for (I) future research on pathology foundation models and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision-making purposes.
Primary Intended Use
The primary intended use is to support AI researchers reproducing and building on top of this work. GigaPath should be helpful for exploring pre-training, and encoding of digital pathology slides data.
Out-of-Scope Use
Any deployed use case of the model --- commercial or otherwise --- is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed use cases.
Usage and License Notices
The model is not intended or made available for clinical use as a medical device, clinical support, diagnostic tool, or other technology intended to be used in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions. The model is not designed or intended to be a substitute for professional medical advice, diagnosis, treatment, or judgment and should not be used as such. All users are responsible for reviewing the output of the developed model to determine whether the model meets the user’s needs and for validating and evaluating the model before any clinical use.
Acknowledgements
We would like to express our gratitude to the authors and developers of the exceptional repositories that this project is built upon: DINOv2, MAE, Timm, and TorchScale. Their contributions have been invaluable to our work.
Citation
If you find this model useful for your research and applications, please cite using this BibTeX:
@article{xu2024gigapath,
title={A whole-slide foundation model for digital pathology from real-world data},
author={Xu, Hanwen and Usuyama, Naoto and Bagga, Jaspreet and Zhang, Sheng and Rao, Rajesh and Naumann, Tristan and Wong, Cliff and Gero, Zelalem and González, Javier and Gu, Yu and Xu, Yanbo and Wei, Mu and Wang, Wenhui and Ma, Shuming and Wei, Furu and Yang, Jianwei and Li, Chunyuan and Gao, Jianfeng and Rosemon, Jaylen and Bower, Tucker and Lee, Soohee and Weerasinghe, Roshanthi and Wright, Bill J. and Robicsek, Ari and Piening, Brian and Bifulco, Carlo and Wang, Sheng and Poon, Hoifung},
journal={Nature},
year={2024},
publisher={Nature Publishing Group UK London}
}
- Downloads last month
- 2