zhihuanglab/Haiku-Bi

Haiku(Bi) — alternative bi-modal (CODEX + H&E) version of Haiku

Haiku(Bi) is the bi-modal sibling of Haiku. It uses the same encoders (VirTues-style CODEX/mIF encoder initialised from ESM marker embeddings; MUSK H&E encoder with the last two blocks tuned), the same projection-head design, the same paired training data and the same training schedule (25 epochs) as Haiku — the only difference is the objective: Haiku(Bi) is trained with only the H&E ↔ mIF (CODEX) contrastive term, with no text modality.

To our knowledge, it is among the first models to align H&E histology and spatial proteomics through contrastive pretraining at this scale (26.7M paired H&E / CODEX patches).

What it supports

• image-to-image retrieval (H&E → mIF and mIF → H&E) in the shared embedding space
• H&E-only biomarker inference (e.g. retrieving / predicting mIF profiles from H&E alone)

What it does not support

• text queries / zero-shot text prompts. There is no text encoder in this bundle; calling text encoding raises an error. Use the tri-modal zhihuanglab/Haiku for text.

Contents

• haiku_state_dict.pt — model weights (CODEX + H&E encoders + projections; no text encoder)
• config.json — architecture config + marker lists ("use_text": false, "modalities": ["codex", "he"])
• esm_embeddings/ — per-biomarker ESM embeddings (also embedded in state_dict; kept here for downstream use)
• vocab.pkl — marker vocabulary

Quick start

from models import Haiku   # from the Haiku GitHub repo (src/ on sys.path)

model, tokenizer, marker_embedding = Haiku.from_pretrained(
    "zhihuanglab/Haiku-Bi",
    device="cuda",
    token="hf_...",  # omit if HF_TOKEN / hf auth login is set
)
model.eval()          # tokenizer is None for Haiku(Bi)

# H&E patches: float tensor B x 3 x 384 x 384 (ImageNet-Inception normalised)
he_emb = model.get_features_single_modality({"HandE": he}, modality="he")
# CODEX/mIF: raw images + channel ids, or pre-computed CODEX encoder embeddings
codex_emb = model.get_features_single_modality({"codex_embedding": codex}, modality="codex")

Citation

If you use Haiku(Bi), please cite Haiku (and the upstream models it builds on, MUSK and VirTues).

@article{haiku2026,
  title={Linking Spatial Biology and Clinical Histology via Haiku},
  author={...},
  year={2026}
}