medarc/spectra-virchow-lora

SPECTRA LoRA - Virchow

A LoRA adapter that makes paige-ai/Virchow robust to changes in slide acquisition -- scanner, stain, and centre. It is trained contrastively on registered PLISM tiles, where the same physical tissue location is imaged under many scanner/stain conditions, so the objective is to pull matched conditions of one tile together while pushing different tiles apart. The base model's weights are untouched; only a rank-32 LoRA delta on the attention and MLP projections is learned and released.

Base model

Base repository	paige-ai/Virchow
Pinned revision	19eebc84ae33e79f1b2d866e6ff90ae50e522f9a
Loader	timm
Adapted modules	qkv, attn.proj, mlp.fc1, mlp.fc2 (4 per block x 32 blocks = 128 modules)
Embedding dimension (this readout)	2560

This adapter was trained and evaluated against that exact revision. Applying it to a different revision of the base model is untested.

Seeds

Three independent training seeds are shipped as subfolders. They are not an ensemble -- pick one, or report the spread across all three.

Folder	Original training seed	Selected step	Training run
seed0/	s0	150	genMASK-c50-ms500-virchow-s0-t900-438673
seed1/	s1	150	genMASK-c50-ms500-virchow-s1-t900-438674
seed2/	s2	150	genMASK-c50-ms500-virchow-s2-t900-438675

Usage

import torch
import torchvision.transforms as T
from PIL import Image

import timm
from timm.layers import SwiGLUPacked
from torch.nn import SiLU
from peft import PeftModel

# Load the pinned base weights via timm's hub path. The architecture kwargs below are
# passed explicitly and override anything in the repo config; do NOT build this model
# from the bare architecture name "vit_huge_patch14_224" -- timm's built-in default config
# for that name is a different model.
base = timm.create_model(
    "hf-hub:paige-ai/Virchow@19eebc84ae33e79f1b2d866e6ff90ae50e522f9a",
    pretrained=True,
    img_size=224, init_values=1e-5, mlp_ratio=5.3375,
    mlp_layer=SwiGLUPacked, act_layer=SiLU, dynamic_img_size=True,
    num_classes=0, global_pool="token",
)

model = PeftModel.from_pretrained(base, "medarc/spectra-virchow-lora", subfolder="seed0")
model = model.merge_and_unload()          # fold LoRA into the base weights
model = model.float().eval().cuda()

Preprocessing -- this must match exactly:

tf = T.Compose([
    T.Resize(224, interpolation=T.InterpolationMode.BICUBIC),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])

Forward pass and readout:

img = Image.open("tile.png").convert("RGB")
x = tf(img).unsqueeze(0).cuda()

with torch.inference_mode():
    h = model.forward_features(x)                  # (B, P + N, C)
    feat = torch.cat([h[:, 0], h[:, 1:].mean(1)], dim=-1)   # (B, 2560)

Readout: CLS ++ mean(patch tokens), patch tokens starting at index 1: torch.cat([h[:, 0], h[:, 1:].mean(1)], dim=-1) Embedding dimension: 2560 Token layout: 256 spatial tokens; embed_dim 1280; num_prefix_tokens = 1 (NO register tokens). Patch/dense tokens begin at index 1.

Nothing else from the training run is needed or released. The contrastive projector heads and the pooling head were training-only machinery and are deliberately not part of this repository.

WARNINGS

1. Virchow v1 and Virchow2 are easy to confuse and the mistake is silent.

They share an architecture name (vit_huge_patch14_224), an embedding dimension (1280), a byte-identical adapter size (83,949,648 B), and the same set of adapted modules. They differ in exactly one place that matters at inference: num_prefix_tokens is 1 for Virchow v1 (no registers) and 5 for Virchow2 (1 CLS + 4 registers). Using the wrong slice index does not raise - it silently folds register tokens into, or drops a patch token from, the mean-pooled half of the clsmean readout and corrupts segmentation features. This repo is Virchow v1: slice patch tokens from index 1.

Results

Base model versus base model + this adapter. Values are read from the SPECTRA paper's tables. n = 3 seeds; the interval is mean +/- 2SD across those three seeds, quoted verbatim from the paper's tables (which already report 2SD).

Metric	Base model	+ SPECTRA LoRA (n=3 seeds, mean +/- 2SD)
PathoROB mean robustness index (cross-centre)	0.815	0.890 +/- 0.004
PLISM top-1 retrieval across scanners	0.758	0.997 +/- 0.000
PLISM top-1 retrieval across stains	0.597	0.931 +/- 0.006
HEST mean Pearson r	0.4061	0.4083 +/- 0.0036
CPTAC AUC	0.6608	0.6839 +/- 0.0026

Higher is better on every row. Base-model numbers are single deterministic evaluations of the frozen base and carry no seed spread.

Training

Method	LoRA (PEFT 0.20.0), fp32 tensors
Rank / alpha / dropout	r = 32, alpha = 64 (scaling 2.0), dropout 0.0, bias none
Learning rate	1e-4, weight decay 0.05
Schedule	500 steps total, 200 warmup
Objective	InfoNCE over registered PLISM tiles, split CLS / mean heads (weights 0.5 / 0.5), temperature 0.07
Checkpoint selection	1-SE rule on the PathoROB robustness index curve, applied per seed
Selected steps (this backbone)	150, 150, 150

Checkpoints were written every 50 steps; only the 1-SE-selected step per seed is released. Because selection is per seed, the three seeds of a backbone need not sit at the same step -- and because the selected steps fall inside the 200-step warmup, the released checkpoints are un-annealed.

Full run hyper-parameters are in each seed folder's training_config.json.

training_config.json encoding note. The training code writes same_core_logit_bias_mean as negative infinity, which Python's json module emits as the bare token -Infinity. That token is not valid JSON and is rejected by JSON.parse and most non-Python parsers. In the released files it is encoded as the string "-Infinity" so that the file parses everywhere. Read it back as float("-inf"). This is the only edit made to the training configuration.

Citation

@inproceedings{spectra2026,
  title  = {SPECTRA: cross-acquisition robustness for pathology foundation models},
  author = {TODO: author list},
  year   = {2026},
  note   = {TODO: confirm venue and year before citing -- submitted to NeurIPS 2026},
  url    = {https://github.com/TODO-org/spectra}
}

Code: SPECTRA on GitHub (TODO: fill in the canonical repository URL before publishing).

Licence

See LICENSE in this repository. The adapter weights are MIT-licensed; the base model carries its own separate licence, which you must also comply with.