Lumen

Lumen aligns frozen Virchow2 and BioMedBERT encoders using rank-4 LoRA adapters and projection heads into a shared 512-dimensional space.

This repository contains only the trained alignment parameters (2,984,961 parameters; 12.0 MB). The backbone weights are loaded from their original repositories.

Usage

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("digitalpathologybern/Lumen", trust_remote_code=True).eval()

# images: (B, 3, 224, 224), preprocessed with the Virchow2 transform
tokens = model.tokenizer(["metastatic carcinoma in a lymph node",
                          "benign lymph node"],
                         padding=True, truncation=True, max_length=128,
                         return_tensors="pt")
with torch.no_grad():
    out = model(images=images, tokens=tokens)

img = torch.nn.functional.normalize(out["image_embeds"], dim=-1)
txt = torch.nn.functional.normalize(out["text_embeds"], dim=-1)
logits = out["logit_scale"] * img @ txt.T

images should have shape (B, 3, 224, 224) and be preprocessed using the Virchow2 transform.

encode_image and encode_text can be used to run either encoder separately.

timm, transformers, and peft are required.

Both backbones are downloaded from their own repositories on the first call.

Licence

The 2,984,961 parameters in this repository are released under CC-BY-NC-4.0. The model also loads two third-party backbones, which are not redistributed here and carry their own terms.

Component	Source	Licence
Adapters, projection heads, logit scale	this repository	CC-BY-NC-4.0
Vision encoder	paige-ai/Virchow2	CC-BY-NC-ND-4.0, gated
Text encoder	microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext	MIT

The assembled model is non-commercial, a constraint that comes from Virchow2 and holds whatever these weights say. Read the backbone terms against your intended use rather than reading the tag above as covering the whole.

Adapting these weights further is permitted. Virchow2 additionally forbids derivatives of itself, so fine-tune the alignment, not the vision encoder.