medarc/spectra-openmidnight-lora

SPECTRA LoRA - OpenMidnight

A LoRA adapter that makes SophontAI/OpenMidnight robust to changes in slide acquisition -- scanner, stain, and centre. It is trained contrastively on registered PLISM tiles, where the same physical tissue location is imaged under many scanner/stain conditions, so the objective is to pull matched conditions of one tile together while pushing different tiles apart. The base model's weights are untouched; only a rank-32 LoRA delta on the attention and MLP projections is learned and released.

Base model

Base repository	SophontAI/OpenMidnight
Pinned revision	87189e6674d397a14a5cd342c97b1a1615a185aa
Loader	timm
Adapted modules	qkv, attn.proj, mlp.fc1, mlp.fc2 (4 per block x 40 blocks = 160 modules)
Embedding dimension (this readout)	1536

This adapter was trained and evaluated against that exact revision. Applying it to a different revision of the base model is untested.

Seeds

Three independent training seeds are shipped as subfolders. They are not an ensemble -- pick one, or report the spread across all three.

Folder	Original training seed	Selected step	Training run
seed0/	s0	150	genMASK-c50-ms500-openmidnight-s0-t900-438670
seed1/	s1	150	genMASK-c50-ms500-openmidnight-s1-t900-438671
seed2/	s2	150	genMASK-c50-ms500-openmidnight-s2-t900-438672

Usage

import torch
import torchvision.transforms as T
from PIL import Image

import timm
from peft import PeftModel

# Load the pinned base weights via timm's hub path. The architecture kwargs below are
# passed explicitly and override anything in the repo config; do NOT build this model
# from the bare architecture name "vit_giant_patch14_reg4_dinov2" -- timm's built-in default config
# for that name is a different model.
base = timm.create_model(
    "hf-hub:SophontAI/OpenMidnight@87189e6674d397a14a5cd342c97b1a1615a185aa",
    pretrained=True,
    img_size=224, init_values=1e-5, dynamic_img_size=False,
    num_classes=0, global_pool="token",
)

model = PeftModel.from_pretrained(base, "medarc/spectra-openmidnight-lora", subfolder="seed0")
model = model.merge_and_unload()          # fold LoRA into the base weights
model = model.float().eval().cuda()

Preprocessing -- this must match exactly:

tf = T.Compose([
    T.Resize(256, interpolation=T.InterpolationMode.BICUBIC),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])

Forward pass and readout:

img = Image.open("tile.png").convert("RGB")
x = tf(img).unsqueeze(0).cuda()

with torch.inference_mode():
    h = model.forward_features(x)                  # (B, P + N, C)
    feat = h[:, 0]                                 # (B, 1536)

Readout: CLS token alone: h[:, 0] Embedding dimension: 1536 Token layout: 256 spatial tokens; embed_dim 1536; num_prefix_tokens = 5 (1 CLS + 4 registers). Patch/dense tokens begin at index 5.

Nothing else from the training run is needed or released. The contrastive projector heads and the pooling head were training-only machinery and are deliberately not part of this repository.

WARNINGS

1. TWO different preprocessing transforms, chosen by benchmark. Getting this wrong changes segmentation results materially.

Default path (HEST, PathoROB, CPTAC, and general feature extraction): resolve from the checkpoint config - it carries no crop_pct and no interpolation, so timm's defaults give Resize(256, bicubic) -> CenterCrop(224).

THUNDER only: a squash resize with no crop - Resize((224, 224)) -> ToTensor -> Normalize(ImageNet). THUNDER's own get_openmidnight hand-builds this. Using the generic resize+crop path instead put the base model 6.2 F1 below the published leaderboard on segmentation and roughly 1 point below on k-NN and linear probing.

2. Normalisation is ImageNet, NOT (0.5,)*3.

OpenMidnight is a replication of the Midnight recipe, but it does not share Midnight's normalisation. kaiko-ai/midnight demands (0.5, 0.5, 0.5); OpenMidnight uses ImageNet statistics. The training checkpoint itself stores no normalisation - the ImageNet choice follows the SophontAI model card.

3. dynamic_img_size=False and img_size=224 are both required.

Results

Base model versus base model + this adapter. Values are read from the SPECTRA paper's tables. n = 3 seeds; the interval is mean +/- 2SD across those three seeds, quoted verbatim from the paper's tables (which already report 2SD).

Metric	Base model	+ SPECTRA LoRA (n=3 seeds, mean +/- 2SD)
PathoROB mean robustness index (cross-centre)	0.618	0.879 +/- 0.007
PLISM top-1 retrieval across scanners	0.703	0.948 +/- 0.015
PLISM top-1 retrieval across stains	0.486	0.782 +/- 0.029
HEST mean Pearson r	0.3902	0.4048 +/- 0.0014
CPTAC AUC	0.6561	0.6844 +/- 0.0022

Higher is better on every row. Base-model numbers are single deterministic evaluations of the frozen base and carry no seed spread.

Training

Method	LoRA (PEFT 0.20.0), fp32 tensors
Rank / alpha / dropout	r = 32, alpha = 64 (scaling 2.0), dropout 0.0, bias none
Learning rate	1e-4, weight decay 0.05
Schedule	500 steps total, 200 warmup
Objective	InfoNCE over registered PLISM tiles, split CLS / mean heads (weights 0.5 / 0.5), temperature 0.07
Checkpoint selection	1-SE rule on the PathoROB robustness index curve, applied per seed
Selected steps (this backbone)	150, 150, 150

Checkpoints were written every 50 steps; only the 1-SE-selected step per seed is released. Because selection is per seed, the three seeds of a backbone need not sit at the same step -- and because the selected steps fall inside the 200-step warmup, the released checkpoints are un-annealed.

Full run hyper-parameters are in each seed folder's training_config.json.

training_config.json encoding note. The training code writes same_core_logit_bias_mean as negative infinity, which Python's json module emits as the bare token -Infinity. That token is not valid JSON and is rejected by JSON.parse and most non-Python parsers. In the released files it is encoded as the string "-Infinity" so that the file parses everywhere. Read it back as float("-inf"). This is the only edit made to the training configuration.

Citation

@inproceedings{spectra2026,
  title  = {SPECTRA: cross-acquisition robustness for pathology foundation models},
  author = {TODO: author list},
  year   = {2026},
  note   = {TODO: confirm venue and year before citing -- submitted to NeurIPS 2026},
  url    = {https://github.com/TODO-org/spectra}
}

Code: SPECTRA on GitHub (TODO: fill in the canonical repository URL before publishing).

Licence

See LICENSE in this repository. The adapter weights are MIT-licensed; the base model carries its own separate licence, which you must also comply with.