mkkim032599/HNE2Space

Hierarchical gated multi-task regression models that predict 11 cell-type proportions from a single H&E histopathology patch.

This repository contains the primary model (H-optimus-0 encoder) together with baseline variants used for comparison.

• License: Apache-2.0 (see License and attribution below for the CONCH variant)

---

Available variants

Variant	Encoder	Feature dim	Released weights	File
hoptimus	H-optimus-0 (ViT-g/14)	1536	Full (encoder + head)	hoptimus/model.safetensors
resnet50	ResNet-50	2048	Full (encoder + head)	resnet50/model.safetensors
conch	CONCH ViT-B/16	512	Head only	conch/head.safetensors

Repository contents

Path	Description
modeling.py	Model definitions — required to load any of the weights
preprocess.py	Whole-slide preprocessing: background removal, stain normalization, patching
inference.py	Command-line inference example
sample_data/	Example region and patches for testing (CC BY 4.0, see below)

About the CONCH variant

CONCH is distributed by MahmoodLab under CC-BY-NC-ND 4.0, which prohibits redistribution. Only the task head is released here — the encoder is not included and must be obtained by you.

Two important caveats:

1. You must request access to CONCH yourself (gated, non-commercial research use only) and install the conch package.
2. This head will not reproduce the paper's CONCH results. The reported numbers used an encoder that was fine-tuned jointly with the head; pairing this head with the original frozen CONCH encoder is a different model. It is provided for architecture reference and completeness only.

---

Predicted cell types

Group	Cell types
Lymphoid	CD4, CD8, B, Plasma
Myeloid	Macrophage, DC
Stromal	endothelial, Fibroblast
Tumor	malignant, NormalEpithelial, dead

Predictions are relative quantile values in [0, 1], not absolute cell counts.

---

Installation

pip install torch torchvision timm safetensors huggingface_hub pillow

For whole-slide preprocessing (preprocess.py), additionally:

pip install numpy tifffile scikit-image

For the CONCH variant, additionally:

pip install git+https://github.com/mahmoodlab/CONCH.git

---

Quick start

import torch
from PIL import Image
from huggingface_hub import hf_hub_download
from modeling import load_model, build_transform, predict

REPO_ID = "mkkim032599/HNE2Space"
DEVICE  = "cuda" if torch.cuda.is_available() else "cpu"

weights = hf_hub_download(REPO_ID, filename="hoptimus/model.safetensors")

model     = load_model(weights, backbone="hoptimus", device=DEVICE)
transform = build_transform("hoptimus")

img = Image.open("patch.png").convert("RGB")
x   = transform(img).unsqueeze(0).to(DEVICE)

preds = predict(model, x)          # dict: cell type -> tensor in [0, 1]

for cell_type, value in preds.items():
    print(f"{cell_type}: {value.item():.4f}")

Or from the command line:

1. single image

python inference.py --input patch.png

2. one folder of patches -> one CSV

python inference.py --input patches/ --output results/

4. root of sample folders -> one CSV per sample

python inference.py --input root/ --output results/ --per-sample

ResNet-50 baseline

weights = hf_hub_download(REPO_ID, filename="resnet50/model.safetensors")
model     = load_model(weights, backbone="resnet50", device=DEVICE)
transform = build_transform("resnet50")

CONCH variant (encoder not included)

# 1. Request access at https://huggingface.co/MahmoodLab/CONCH and download
#    the encoder checkpoint yourself.
# 2. Pass its local path:

head = hf_hub_download(REPO_ID, filename="conch/head.safetensors")

model = load_model(
    head,
    backbone="conch",
    conch_checkpoint_path="/path/to/your/CONCH/pytorch_model.bin",
    device=DEVICE,
)
transform = build_transform("conch")

Batch inference

batch = torch.stack([transform(Image.open(p).convert("RGB")) for p in paths])
preds = predict(model, batch.to(DEVICE))   # each value has shape (B,)

---

Input requirements

Patch size	224 x 224
Resolution	approximately 0.27 microns per pixel
Stain	H&E

Normalization differs per encoder — always use build_transform(backbone) rather than ImageNet defaults:

Variant	Mean	Std
hoptimus	(0.707223, 0.578729, 0.703617)	(0.211883, 0.230117, 0.177517)
conch	(0.48145466, 0.4578275, 0.40821073)	(0.26862954, 0.26130258, 0.27577711)
resnet50	(0.485, 0.456, 0.406)	(0.229, 0.224, 0.225)

predict() converts raw outputs from log1p space back to quantile space. If you call model(x) directly, apply from_log1p() yourself.

---

Preprocessing

preprocess.py turns a whole-slide H&E image into model-ready patches:

tif  ->  background removal  ->  Reinhard normalization  ->  patches

1. Background removal. The slide is tiled into small blocks; blocks whose mean HSV saturation falls below a threshold are treated as background and zeroed out. This removes white glass and empty regions before normalization, so background pixels do not distort the colour statistics.

2. Reinhard colour normalization. Stain colour is matched to a reference slide in LAB space, channel by channel. Background (zero) pixels are excluded from the mean/std computation and left untouched. This reduces variation between scanners and staining batches. LAB conversion is chunked so large slides do not exhaust memory.

3. Patch extraction. Patches are cropped on a regular grid and kept only if they pass a saturation floor and a maximum background-fraction check.

Requirements

pip install numpy pillow tifffile scikit-image
pip install h5py        # only for --save-h5

Usage

# single slide
python preprocess.py --input slide.tif --target reference.tif --output out/

# directory of slides
python preprocess.py --input slides/ --pattern "**/Aligned-hne.tif" \
    --target reference.tif --output out/

# patches sized for the model
python preprocess.py --input slide.tif --target reference.tif --output out/ \
    --patch-size 256 --resize 224

--target is a reference slide whose stain appearance you want to match. Use the same reference across your whole cohort — switching references mid-way introduces exactly the batch effect this step is meant to remove.

Key options

Option	Default	Description
--patch-size	256	Crop size in pixels
--stride	= patch size	Step between patches; smaller values overlap
--resize	none	Resize patches (use 224 for the model)
--bg-sat-threshold	0.10	Saturation below which a block is background
--patch-sat-threshold	0.10	Minimum mean saturation to keep a patch
--black-threshold	0.10	Max fraction of background pixels per patch
--no-background-removal	off	Skip stage 1
--no-normalize	off	Skip stage 2
--save-intermediate	off	Also write the normalized slide and a JPEG preview
--save-h5	off	Additionally bundle patches into an HDF5 file

Thresholds are starting points, not tuned values — inspect the output with --save-intermediate and adjust them for your slides.

Resolution

The encoder expects roughly 0.27 microns per pixel. preprocess.py crops in pixel units and does not read scanner MPP metadata, so choose --patch-size and --resize to match your source resolution. For example, a slide at 0.25 um/px needs a 448-pixel crop resized to 224 to land at ~0.5 um/px.

---

Sample data

sample_data/ contains a small H&E region and a few prepared patches so you can verify the pipeline end to end without downloading a full slide.

# run the model on a prepared patch
python inference.py sample_data/patches/patch_00000.png --backbone hoptimus

# or start from the region crop
python preprocess.py --input sample_data/region.tif \
    --target sample_data/region.tif --output my_patches/ --resize 224

These images derive from a publicly available 10x Genomics dataset licensed under CC BY 4.0, cropped and tiled for this repository. See sample_data/README.md for the dataset link and full attribution.

The sample images are distributed under CC BY 4.0 (inherited from the source dataset), not under the Apache-2.0 licence covering the code and weights.

---

Architecture

encoder                       -> feature_dim  (1536 / 2048 / 512)
  shared trunk                -> 256          (via 512 for hoptimus)
  lineage branches      256   -> 128          (lymphoid / myeloid / stromal / tumor)
  per-task gating       128   -> 128          (sigmoid attention gate)
  regression heads      128   -> 1            (one per cell type)

Each cell type is routed through its lineage branch, then through a task-specific gate before its regression head.

If a checkpoint was trained with different architecture settings, pass them explicitly — the defaults must match how the weights were trained:

model = load_model(weights, backbone="resnet50", input_proj_dim=None)

---

Limitations and intended use

• Research use only. Not a medical device; not validated for clinical or diagnostic use. Performance on unseen tissue types, scanners, or staining protocols may degrade.
• Outputs are relative quantile values, not absolute cell counts or densities.
• Accuracy varies by cell type; rarer populations are predicted less reliably. Consult the evaluation table before relying on any individual cell type.

---

License and attribution

The code and the hoptimus / resnet50 weights are released under the Apache License 2.0.

The hoptimus weights are a derivative work of H-optimus-0 (Bioptimus), also Apache-2.0. The encoder weights have been modified (fine-tuned) from the original release. The original copyright notice is retained in NOTICE.

• H-optimus-0: https://huggingface.co/bioptimus/H-optimus-0
• ResNet-50 (torchvision, ImageNet-pretrained initialisation)

---

Citation

@software{hoptimus0,
  author = {Saillard, Charlie and Jenatton, Rodolphe and Llinares-L{\'o}pez, Felipe
            and Mariet, Zelda and Cahan{\'e}, David and Durand, Eric and Vert, Jean-Philippe},
  title  = {H-optimus-0},
  year   = {2024},
  url    = {https://github.com/bioptimus/releases/tree/main/models/h-optimus/v0}
}

@article{lu2024conch,
  title   = {A visual-language foundation model for computational pathology},
  author  = {Lu, Ming Y and Chen, Bowen and Williamson, Drew F K and others},
  journal = {Nature Medicine},
  year    = {2024}
}