sywang/TPIPS-EarlyFusion-Qwen3VL-8B

TPIPS — Early Fusion (R1/R2 register tokens)

Text-conditioned perceptual image similarity, built on Qwen/Qwen3-VL-Embedding-8B. This repo holds the early_fusion checkpoint (one of three TPIPS models, each in its own repo — see the table at the bottom). Code and full docs: https://github.com/PeterWang512/TPIPS.

Early fusion. (img0, img1, text) are encoded jointly with two register tokens R1 / R2 under a block-sparse attention mask. The pairwise score is cos(R1, R2) (symmetrized over both orderings) (higher = more similar). Odd-one-out probabilities are a softmax over the three "other-pair" scores divided by the temperature; 2AFC compares the two reference-candidate scores. The pairwise score is the model's raw output (temperature is applied at the probability step).

Property	Value
Base model	Qwen/Qwen3-VL-Embedding-8B
Pairwise score	cos(R1, R2) (symmetrized over both orderings)
Fine-tuning	LoRA (r=64, α=128) on the LLM layers; vision tower frozen
Register tokens	2 (R1 sees text+img0; R2 sees text+img1)
Temperature	0.05 (applied at the probability step)
Prompt X	Measure the similarity between the two images based on X.

Usage

pip install tpips   # then install flash-attn; see the GitHub README

from huggingface_hub import snapshot_download
from tpips import load_model
from PIL import Image

ckpt = snapshot_download("sywang/TPIPS-EarlyFusion-Qwen3VL-8B")
model = load_model(ckpt, device="cuda")   # auto-detects the model class

a = Image.open("a.jpg").convert("RGB")
b = Image.open("b.jpg").convert("RGB")
model.similarity(a, b, factor="lighting")  # higher = more similar

flash-attn is required (the model runs with flash-attention-2); see the Installation guide for an easy matching-wheel install.

The TPIPS models

Model	Repo
Embedding (late fusion)	sywang/TPIPS-Embed-Qwen3VL-8B
Early Fusion (R1/R2 register tokens)	sywang/TPIPS-EarlyFusion-Qwen3VL-8B
Activation Distance	sywang/TPIPS-ActDiff-Qwen3VL-8B