hunarbatra/vlm_scope

MMDiff — Multimodal SAE checkpoints

Text-only-masked SAEs trained on the residual streams of two multimodal LLMs, used by the MMDiff multimodal model-diffing pipeline (code: hunarbatra/MMDiff).

Contents

paligemma2/jumprelu_text_only/ — 26 layers

JumpReLU SAEs (width 16,384) trained on google/paligemma2-3b-mix-448 language-model residual stream, layers 0–25, with text-only masking (image-token positions are excluded from the training loss). Diffed against the Gemma-Scope base SAEs (google/gemma-scope-2b-pt-res, width_16k).

File: text-only_layer_{L}.pt with keys W_enc, b_enc, W_dec, b_dec, threshold.

llava_more/topk_text_only/ — 32 layers

TopK SAEs (k=50, width 32,768) trained on aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning residual stream, layers 0–31, with text-only masking. Diffed against the LLaMA-Scope base SAEs (fnlp/Llama3_1-8B-Base-LXR-8x, sae-lens release llama_scope_lxr_8x).

File: text-only_layer_{L}.pt with keys W_enc (4096×32768), b_enc (32768), W_dec (32768×4096), b_dec (4096).

Loading

import torch
from huggingface_hub import hf_hub_download

p = hf_hub_download("hunarbatra/vlm_scope", "paligemma2/jumprelu_text_only/text-only_layer_13.pt")
state = torch.load(p, map_location="cpu")

See the MMDiff repo for the full pipeline (training, adapted-feature selection, task-specific discovery, ablation, steering).