medarc/spectra-midnight-12k-lora

SPECTRA LoRA - Midnight-12k

A LoRA adapter that makes kaiko-ai/midnight robust to changes in slide acquisition -- scanner, stain, and centre. It is trained contrastively on registered PLISM tiles, where the same physical tissue location is imaged under many scanner/stain conditions, so the objective is to pull matched conditions of one tile together while pushing different tiles apart. The base model's weights are untouched; only a rank-32 LoRA delta on the attention and MLP projections is learned and released.

Base model

Base repository	kaiko-ai/midnight
Pinned revision	adc6b15679c981cce6f9b018bbad09d16eeeda9f
Loader	Hugging Face transformers
Adapted modules	query, key, value, attention.output.dense, mlp.weights_in, mlp.weights_out (6 per block x 40 blocks = 240 modules). Note the FFN leaves are weights_in/weights_out, NOT fc1/fc2, because the FFN is SwiGLU.
Embedding dimension (this readout)	3072

This adapter was trained and evaluated against that exact revision. Applying it to a different revision of the base model is untested.

Seeds

Three independent training seeds are shipped as subfolders. They are not an ensemble -- pick one, or report the spread across all three.

Folder	Original training seed	Selected step	Training run
seed0/	s0	150	genMASK-c50-lr1e-4-kl0-ms500-midnight-s0-t900-399166
seed1/	s1	100	genMASK-c50-lr1e-4-kl0-ms500-midnight-s1-t900-407566
seed2/	s3	100	genMASK-c50-ms500-midnight-s3-t900-436609.r5

Seed label remapping. The published folder seed2/ was trained with seed = 3, not 2 (there is no s2 run for this backbone). The folder names are normalised to seed0/seed1/seed2 for consistency across the release; the original training seed value is recorded in each folder's training_config.json under "seed".

Usage

import torch
import torchvision.transforms as T
from PIL import Image

from transformers import AutoModel
from peft import PeftModel

base = AutoModel.from_pretrained(
    "kaiko-ai/midnight",
    revision="adc6b15679c981cce6f9b018bbad09d16eeeda9f",
)
model = PeftModel.from_pretrained(base, "medarc/spectra-midnight-12k-lora", subfolder="seed0")
model = model.merge_and_unload()          # fold LoRA into the base weights
model = model.float().eval().cuda()

Preprocessing -- this must match exactly:

tf = T.Compose([
    T.Resize(224, interpolation=T.InterpolationMode.BILINEAR),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),
])

Forward pass and readout:

img = Image.open("tile.png").convert("RGB")
x = tf(img).unsqueeze(0).cuda()

with torch.inference_mode():
    h = model(pixel_values=x).last_hidden_state    # (B, 1 + N, C)
    feat = torch.cat([h[:, 0], h[:, 1:].mean(1)], dim=-1)   # (B, 3072)

Readout: CLS ++ mean(patch tokens): torch.cat([h[:, 0], h[:, 1:].mean(1)], dim=-1) Embedding dimension: 3072 Token layout: 256 spatial tokens; embed_dim 1536. Patch/dense tokens begin at index 1.

Nothing else from the training run is needed or released. The contrastive projector heads and the pooling head were training-only machinery and are deliberately not part of this repository.

WARNINGS

1. Normalisation is (0.5, 0.5, 0.5), not ImageNet.

kaiko-ai/midnight ships no preprocessor_config.json, so feeding ImageNet statistics will neither crash nor emit a warning - it will just silently degrade every downstream number. Use (0.5,)*3.

2. Resize interpolation is bilinear, not bicubic.

This matches the reference transforms.Resize(224) call (torchvision's default is bilinear). Every other backbone in this release uses bicubic.

Results

Base model versus base model + this adapter. Values are read from the SPECTRA paper's tables. n = 3 seeds; the interval is mean +/- 2SD across those three seeds, quoted verbatim from the paper's tables (which already report 2SD).

Metric	Base model	+ SPECTRA LoRA (n=3 seeds, mean +/- 2SD)
PathoROB mean robustness index (cross-centre)	0.759	0.908 +/- 0.005
PLISM top-1 retrieval across scanners	0.752	0.991 +/- 0.005
PLISM top-1 retrieval across stains	0.560	0.883 +/- 0.036
HEST mean Pearson r	0.3952	0.4122 +/- 0.0022
CPTAC AUC	0.6643	0.6898 +/- 0.0016

Higher is better on every row. Base-model numbers are single deterministic evaluations of the frozen base and carry no seed spread.

Training

Method	LoRA (PEFT 0.20.0), fp32 tensors
Rank / alpha / dropout	r = 32, alpha = 64 (scaling 2.0), dropout 0.0, bias none
Learning rate	1e-4, weight decay 0.05
Schedule	500 steps total, 200 warmup
Objective	InfoNCE over registered PLISM tiles, split CLS / mean heads (weights 0.5 / 0.5), temperature 0.07
Checkpoint selection	1-SE rule on the PathoROB robustness index curve, applied per seed
Selected steps (this backbone)	150, 100, 100

Checkpoints were written every 50 steps; only the 1-SE-selected step per seed is released. Because selection is per seed, the three seeds of a backbone need not sit at the same step -- and because the selected steps fall inside the 200-step warmup, the released checkpoints are un-annealed.

Full run hyper-parameters are in each seed folder's training_config.json.

training_config.json encoding note. The training code writes same_core_logit_bias_mean as negative infinity, which Python's json module emits as the bare token -Infinity. That token is not valid JSON and is rejected by JSON.parse and most non-Python parsers. In the released files it is encoded as the string "-Infinity" so that the file parses everywhere. Read it back as float("-inf"). This is the only edit made to the training configuration.

Citation

@inproceedings{spectra2026,
  title  = {SPECTRA: cross-acquisition robustness for pathology foundation models},
  author = {TODO: author list},
  year   = {2026},
  note   = {TODO: confirm venue and year before citing -- submitted to NeurIPS 2026},
  url    = {https://github.com/TODO-org/spectra}
}

Code: SPECTRA on GitHub (TODO: fill in the canonical repository URL before publishing).

Licence

See LICENSE in this repository. The adapter weights are MIT-licensed; the base model carries its own separate licence, which you must also comply with.