iSight-slide: image-level assessment of immunohistochemistry staining

The image-level model of iSight, a system for automated assessment of immunohistochemistry (IHC) images and protein staining patterns. This repository carries the trained checkpoint and the training and inference code.

🔗 Links

• Code repository: github.com/zhihuanglab/iSight
• Training dataset: nirschl-lab/hpa10m
• Companion models: zhihuanglab/iSight-cell (per-cell staining), zhihuanglab/iSight-target (target-cell selection)

🎯 Prediction tasks

The model predicts five attributes of an IHC image at once:

Task	Classes	Labels
Staining intensity	4	negative, weak, moderate, strong
Staining location	4	none, cytoplasmic/membranous, nuclear, cytoplasmic/membranous,nuclear
Staining quantity	4	none, <25%, 25%-75%, >75%
Tissue type	58	human tissue types
Malignancy	2	normal, cancer

🧠 Model

CLIP ViT-L/14-336 patch encoder over all 336 px tissue patches of an image. Every patch contributes all 576 of its ViT tokens; a gated attention module scores each token position and softmaxes across patches at that position, so pooling is per token rather than per patch. The pooled representation is the mean over tokens. Two conditioning signals are added to it: a text (context) branch encoding the query (tissue, diagnosis, gene), applied with dropout during training and off at inference, and a 39-way cell-type embedding. Five linear heads produce the outputs above. Model version v3_all_tokens.

📦 Files

config.json                               model configuration (read this first; see Downloading)
checkpoints/iSight-slide.pth   trained weights (model state dict only)
code/
  model/patch_encoder_with_clam.py        encoder, all-token gated attention, conditioning, heads
  dataset/hpadataset.py                   HPA10M MIL dataset, tissue-mask patching
  train.py                                training (DDP, resumable)
  config/config.ini                       the configuration the checkpoint was trained under
  scripts/inference.py                    image-level inference
  tissue.py                               tissue mask
requirements.txt

⬇️ Downloading

from huggingface_hub import snapshot_download
local = snapshot_download("zhihuanglab/iSight-slide")        # config.json + checkpoint + code

or a single file:

from huggingface_hub import hf_hub_download
cfg  = hf_hub_download("zhihuanglab/iSight-slide", "config.json")
ckpt = hf_hub_download("zhihuanglab/iSight-slide", "checkpoints/iSight-slide.pth")

Please fetch config.json alongside the checkpoint: it carries the model settings the code reads, and it is the file the Hub uses to count downloads of this repository.

🚀 Running

pip install -r requirements.txt
export ISIGHT_DATA_ROOT=/path/to/hpa10m          # metadata, RLE masks, images
cd code
python train.py --config config/config.ini

Data locations are environment variables, not hard-coded paths:

variable	what
ISIGHT_DATA_ROOT	root for the defaults below
ISIGHT_TRAIN_META / ISIGHT_TEST_META	HPA10M split metadata (feather)
ISIGHT_RLE_DIR / ISIGHT_RLE_INDEX	RLE tissue masks and their index
ISIGHT_IMAGE_DIR	images, only for the simple_downsample version
SCHEDULER_PER_EPOCH=1	step the LR scheduler per epoch instead of per batch

The released configuration uses batch_size = 1, which is what the checkpoint was trained with.

📧 Contact

Zhi Huang — zhi.huang@pennmedicine.upenn.edu