zer0int/CLIP-ViT-L-14-Universal-VPT-ReadNull-Token

CLIP-ViT-L-14-Universal-VPT-ReadNull-Token

Tl;dr: A tiny read_null_token.safetensors (1024-D, 5KB). Appending it to an OpenAI CLIP-L Vision Transformer (or any fine-tune thereof) greatly improves robustness to 'reading' text in images. No additional training needed!
Also includes a full model.safetensors for your convenience (/use with RN token).

---

ℹ️ This repo contains a ReadNull (RN) token that most closely resembles Visual Prompt Tuning.
Unlike conventional prompt tuning, this standalone ReadNull (RN) token can be transplanted into compatible OpenAI pretrained or fine-tuned CLIP ViT-L/14 and ViT-L/14@336px models without further optimization.
🔍 The Read-Null (RN) token is a universal learned synthetic K/V source that appears to accesses a highly conserved control interface in OAI CLIP-L. This perturbs the internal computation in the second half of the transformer in a targeted way, ultimately leading to typographic attack-relevant directions to be suppressed in the final embedding - yielding greatly improved robustness.

Made with love & mechanistic interpretability. Trained on a cross-attention bridge.

---

RN TOKEN

read_null_token.safetensors:
• Is a standalone learned Read-Null (RN) token for OpenAI CLIP ViT-L/14 lineage models.
• Contains one 1024-dimensional tensor.
• The rn_adapter.py appends it to the vision token sequence immediately before vision block 13 (zero-based).
• The code uses an explicit PyTorch forward pre-hook.
•💡 It does not require trust_remote_code=True, and does not require any training.
•💡 See 👉 below for example code and insert any (your) OAI CLIP ViT-L/14 fine-tune!

🔍 Fun fact: Touch & go is enough. Inserting the RN token before B13 and removing it after block 13 essentially preserves the entire effect. Example:

RN in place	Logit Margin
RN off	+3.446
RN through B13 only	+8.269
RN persistent to B23	+8.282

BASE MODEL

This repository also contains a fully fine-tuned CLIP ViT-L/14 model.safetensors.
It is the continuation of my GmP fine-tune and serves as the backbone of the full cross-attention model (coming soon; will allow you to flip between ignoring text vs. reading all the text in an image - using a special text switch token!).
Finally, the rn_model_text_encoder.safetensors can be used for text-to-image generative AI, e.g. in ComfyUI.

---

👉 Click here to expand example code (RN token + ZS Typographic Attack)

from __future__ import annotations

import json
import argparse
import random
import textwrap
from dataclasses import dataclass
from pathlib import Path, PureWindowsPath
from typing import Any
import numpy as np
import torch

from datasets import load_dataset
from transformers import CLIPModel, CLIPProcessor
from huggingface_hub import hf_hub_download
from safetensors import safe_open
from safetensors.torch import load_file


RN_ADAPTER_REPO_DIR = "zer0int/CLIP-ViT-L-14-Universal-VPT-ReadNull-Token" # or path with local read_null_token.safetensors

DEFAULT_MODELS = (
    "openai/clip-vit-large-patch14",
    "zer0int/CLIP-GmP-ViT-L-14",
    "zer0int/CLIP-ViT-L-14-Universal-VPT-ReadNull-Token",
) # Models to apply the RN token to.

"""
NOTE!
Models must be OpenAI CLIP ViT-L/14 or ViT-L/14@336px (or a fine-tune thereof).
Entirely different models (e.g. OpenCLIP, AppleCLIP) have different pretraining
distributions etc., and *this* particular RN token won't work for improving them.

"""



# ==============
#  RN ADAPTER
# ==============

RN_FILENAME = "read_null_token.safetensors"
SUPPORTED_IMAGE_SIZES = (224, 336)


def _vision_model(model: Any):
    candidate = getattr(model, "vision_model", model)
    if hasattr(candidate, "vision_model"):
        candidate = candidate.vision_model
    if not all(hasattr(candidate, name) for name in ("embeddings", "encoder")):
        raise TypeError(
            "Expected a CLIP-like model exposing vision_model.embeddings and "
            "vision_model.encoder"
        )
    return candidate


def _architecture(vision_model: Any) -> dict[str, int]:
    config = vision_model.config
    image_size = config.image_size
    patch_size = config.patch_size
    if isinstance(image_size, (list, tuple)):
        if len(set(image_size)) != 1:
            raise ValueError(f"Only square CLIP inputs are supported, got {image_size}")
        image_size = image_size[0]
    if isinstance(patch_size, (list, tuple)):
        if len(set(patch_size)) != 1:
            raise ValueError(f"Only square patches are supported, got {patch_size}")
        patch_size = patch_size[0]
    return {
        "image_size": int(image_size),
        "patch_size": int(patch_size),
        "vision_width": int(config.hidden_size),
        "vision_layers": int(config.num_hidden_layers),
        "vision_heads": int(config.num_attention_heads),
    }


def validate_vit_l_14(vision_model: Any) -> dict[str, int]:
    """Require the OpenAI-style ViT-L/14 tensor architecture at 224 or 336 px."""
    actual = _architecture(vision_model)
    expected = {
        "patch_size": 14,
        "vision_width": 1024,
        "vision_layers": 24,
        "vision_heads": 16,
    }
    failures = [
        f"{key}={actual[key]} expected={value}"
        for key, value in expected.items()
        if actual[key] != value
    ]
    if actual["image_size"] not in SUPPORTED_IMAGE_SIZES:
        failures.append(
            f"image_size={actual['image_size']} expected one of {SUPPORTED_IMAGE_SIZES}"
        )
    if failures:
        raise ValueError("RN token requires ViT-L/14: " + "; ".join(failures))
    return actual


def _resolve_rn_file(path_or_repo_id: str | Path, revision: str | None) -> Path:
    path = Path(path_or_repo_id).expanduser()
    if path.is_file():
        return path.resolve()
    if path.is_dir():
        candidate = path / RN_FILENAME
        if not candidate.is_file():
            raise FileNotFoundError(candidate)
        return candidate.resolve()
    return Path(
        hf_hub_download(
            repo_id=str(path_or_repo_id), filename=RN_FILENAME, revision=revision
        )
    )


def _metadata(path: Path) -> dict[str, str]:
    with safe_open(path, framework="pt", device="cpu") as handle:
        return dict(handle.metadata() or {})


def _metadata_int(metadata: dict[str, str], key: str, fallback: int) -> int:
    value = metadata.get(key)
    return int(value) if value is not None else int(fallback)


def read_null_status(model: Any) -> str | None:
    """Return ``"adapter"`` when this helper has RN active, else ``None``."""
    vision = _vision_model(model)
    if getattr(vision, "_rn_adapter_handle", None) is not None:
        return "adapter"
    return None

def has_read_null_token(model: Any) -> bool:
    """Return whether an RN token is active in the loaded model."""
    return read_null_status(model) is not None


def apply_read_null_token(
    model: Any,
    token_path_or_repo_id: str | Path,
    *,
    revision: str | None = None,
    debug: bool = False,
):
    """Append a learned RN token immediately before its checkpoint-defined ViT block."""
    vision = _vision_model(model)
    actual = validate_vit_l_14(vision)
    status = read_null_status(model)
    if status is not None:
        raise RuntimeError(f"An RN token is already active ({status})")

    token_path = _resolve_rn_file(token_path_or_repo_id, revision)
    tensors = load_file(str(token_path), device="cpu")
    if "read_null_token" not in tensors:
        raise KeyError(f"{token_path} has no 'read_null_token' tensor")
    token = tensors["read_null_token"]
    if tuple(token.shape) not in ((actual["vision_width"],), (1, actual["vision_width"])):
        raise ValueError(
            f"RN tensor shape {tuple(token.shape)} does not match vision width "
            f"{actual['vision_width']}"
        )
    token = token.reshape(actual["vision_width"])

    metadata = _metadata(token_path)
    insert_block = _metadata_int(metadata, "read_null_insert_block", 13)
    metadata_width = _metadata_int(metadata, "vision_width", actual["vision_width"])
    metadata_size = _metadata_int(metadata, "image_size", actual["image_size"])
    if metadata_width != actual["vision_width"]:
        raise ValueError(
            f"RN metadata vision_width={metadata_width} conflicts with model "
            f"vision_width={actual['vision_width']}"
        )
    if metadata_size not in SUPPORTED_IMAGE_SIZES:
        raise ValueError(
            f"RN metadata image_size={metadata_size} is unsupported; expected one of "
            f"{SUPPORTED_IMAGE_SIZES}"
        )
    if not 0 <= insert_block < actual["vision_layers"]:
        raise ValueError(f"RN insertion block {insert_block} is outside the vision stack")

    device = vision.embeddings.class_embedding.device
    dtype = vision.embeddings.class_embedding.dtype
    vision.register_parameter(
        "read_null_token", torch.nn.Parameter(token.to(device=device, dtype=dtype))
    )
    vision.read_null_insert_block = insert_block

    def append_rn(_module, args):
        if not args:
            raise RuntimeError("CLIP encoder layer received no hidden states")
        hidden_states = args[0]
        rn = vision.read_null_token.to(
            device=hidden_states.device, dtype=hidden_states.dtype
        ).view(1, 1, -1)
        rn = rn.expand(hidden_states.shape[0], 1, -1)
        return (torch.cat((hidden_states, rn), dim=1), *args[1:])

    layer = vision.encoder.layers[insert_block]
    vision._rn_adapter_handle = layer.register_forward_pre_hook(append_rn)
    vision._rn_adapter_source = str(token_path)
    vision._rn_adapter_owns_parameter = True

    if debug:
        print("[RN adapter] model architecture:", json.dumps(actual, sort_keys=True))
        print(f"[RN adapter] token insertion: before zero-based block {insert_block}")
        print(
            f"[RN adapter] token: {token_path} | shape={tuple(token.shape)} | "
            f"fp32_l2={token.float().norm().item():.8f}"
        )
        print(
            "[RN adapter] dimensions: OK; "
            "for compatibility: ensure OpenAI CLIP ViT-L/14-lineage weights"
        )
    return model


def ensure_read_null_token(
    model: Any,
    token_path_or_repo_id: str | Path,
    *,
    revision: str | None = None,
    debug: bool = False,
):
    """Leave an RN adapter already installed by this helper, otherwise attach it."""
    status = read_null_status(model)
    if status is not None:
        if debug:
            print(f"[RN adapter] RN already active ({status}); leaving it unchanged.")
        return model
    return apply_read_null_token(
        model, token_path_or_repo_id, revision=revision, debug=debug
    )


def remove_read_null_token(model: Any):
    """Remove an RN token installed by this helper."""
    vision = _vision_model(model)
    handle = getattr(vision, "_rn_adapter_handle", None)
    if handle is None:
        return model

    handle.remove()
    vision._rn_adapter_handle = None
    vision._rn_adapter_source = None
    vision.read_null_insert_block = None
    if getattr(vision, "_rn_adapter_owns_parameter", False) and hasattr(
        vision, "read_null_token"
    ):
        delattr(vision, "read_null_token")
    vision._rn_adapter_owns_parameter = False
    return model


def load_clip_with_rn(
    base_model_id_or_path: str | Path,
    token_path_or_repo_id: str | Path | None = None,
    *,
    revision: str | None = None,
    debug: bool = False,
    **from_pretrained_kwargs,
):
    """Load a stock HF CLIP checkpoint and activate RN.

    When ``token_path_or_repo_id`` is omitted, RN is loaded from the same repository
    as the checkpoint. This is the intended loader for the bundled base + RN release.
    """
    from transformers import CLIPModel

    model = CLIPModel.from_pretrained(
        str(base_model_id_or_path), revision=revision, **from_pretrained_kwargs
    )
    token_source = token_path_or_repo_id or base_model_id_or_path
    return ensure_read_null_token(
        model, token_source, revision=revision, debug=debug
    )

# ===============================================================================



# ================
#  ZS Typo Attack
# ================

PROMPT = "a photo of a {word}"
DEFAULT_SEED = 20260829

SUBSET_ORDER = (
    ("SCAM", "NoSCAM"),
    ("SCAM", "SCAM"),
    ("SCAM", "SynthSCAM"),
    ("RTA", "NoRTA"),
    ("RTA", "RTA"),
    ("RTA", "SynthRTA"),
)


@dataclass
class PairSample:
    image: Any
    correct_label: str
    distractor_label: str


@dataclass
class SubsetStats:
    count: int
    accuracy: float
    mean_logit_margin: float


def configure_reproducibility(seed: int) -> None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False


def _display_model_reference(model_reference: str) -> str:
    raw = str(model_reference)
    path = Path(raw).expanduser()
    windows_path = "\\" in raw
    if path.is_dir() or raw in {".", ".."}:
        resolved = path.resolve()
        return f"{resolved.name} ({raw})"
    if windows_path:
        return PureWindowsPath(raw.rstrip("\\/")).name
    return raw.rstrip("/")


def load_scam_samples() -> dict[str, list[PairSample]]:
    print("[benchmark] Loading BLISS-e-V/SCAM ...")
    dataset = load_dataset("BLISS-e-V/SCAM", split="train")
    buckets = {name: [] for name in ("NoSCAM", "SCAM", "SynthSCAM")}
    for entry in dataset:
        sample_id = str(entry["id"])
        variant = next((name for name in buckets if sample_id.startswith(name)), None)
        if variant is None:
            continue
        buckets[variant].append(
            PairSample(
                image=entry["image"],
                correct_label=str(entry["object_label"]),
                distractor_label=str(entry["attack_word"]),
            )
        )
    print(
        "[benchmark] SCAM subsets: "
        + ", ".join(f"{name}={len(samples)}" for name, samples in buckets.items())
    )
    return buckets


def load_rta_samples() -> dict[str, list[PairSample]]:
    print("[benchmark] Loading zer0int/RTA-100-Triplet ...")
    dataset = load_dataset("zer0int/RTA-100-Triplet", split="train")
    buckets = {name: [] for name in ("NoRTA", "RTA", "SynthRTA")}
    for entry in dataset:
        variant = str(entry["type"])
        if variant not in buckets:
            continue
        buckets[variant].append(
            PairSample(
                image=entry["image"],
                correct_label=str(entry["object_label"]),
                distractor_label=str(entry["attack_word"]),
            )
        )
    print(
        "[benchmark] RTA subsets: "
        + ", ".join(f"{name}={len(samples)}" for name, samples in buckets.items())
    )
    return buckets


def _feature_tensor(output: Any) -> torch.Tensor:
    if torch.is_tensor(output):
        return output
    for name in ("pooler_output", "image_embeds", "text_embeds"):
        value = getattr(output, name, None)
        if torch.is_tensor(value):
            return value
    if isinstance(output, (tuple, list)) and output and torch.is_tensor(output[0]):
        return output[0]
    raise TypeError(f"Cannot locate feature tensor in {type(output)!r}")


def _encode_texts(
    model: Any,
    processor: Any,
    labels: list[str],
    device: torch.device,
) -> torch.Tensor:
    prompts = [PROMPT.format(word=label) for label in labels]
    inputs = processor(text=prompts, padding=True, return_tensors="pt")
    inputs = {key: value.to(device) for key, value in inputs.items()}
    with torch.inference_mode():
        features = _feature_tensor(model.get_text_features(**inputs))
    return torch.nn.functional.normalize(features.float(), dim=-1).cpu()


def _encode_images(
    model: Any,
    processor: Any,
    images: list[Any],
    device: torch.device,
    batch_size: int,
) -> torch.Tensor:
    chunks: list[torch.Tensor] = []
    for start in range(0, len(images), batch_size):
        inputs = processor(images=images[start : start + batch_size], return_tensors="pt")
        pixel_values = inputs["pixel_values"].to(device)
        with torch.inference_mode():
            features = _feature_tensor(model.get_image_features(pixel_values=pixel_values))
        chunks.append(torch.nn.functional.normalize(features.float(), dim=-1).cpu())
    return torch.cat(chunks) if chunks else torch.empty((0, 0), dtype=torch.float32)


def _logit_scale(model: Any) -> float:
    value = getattr(model, "logit_scale", None)
    if value is None:
        raise AttributeError("Model has no CLIP-style logit_scale parameter")
    return float(value.detach().float().exp().cpu())


def _evaluate_binary(
    samples: list[PairSample],
    image_features: torch.Tensor,
    text_features: torch.Tensor,
    label_index: dict[str, int],
    logit_scale: float,
) -> SubsetStats:
    correct = 0
    margins: list[float] = []
    for sample, image_feature in zip(samples, image_features):
        object_logit = (
            float(image_feature @ text_features[label_index[sample.correct_label]])
            * logit_scale
        )
        attack_logit = (
            float(image_feature @ text_features[label_index[sample.distractor_label]])
            * logit_scale
        )
        margin = object_logit - attack_logit
        margins.append(margin)
        if margin >= 0.0:
            correct += 1

    count = len(samples)
    return SubsetStats(
        count=count,
        accuracy=correct / count if count else 0.0,
        mean_logit_margin=sum(margins) / count if count else 0.0,
    )


def evaluate_variant(
    model: Any,
    processor: Any,
    datasets: dict[str, dict[str, list[PairSample]]],
    text_features: torch.Tensor,
    label_index: dict[str, int],
    device: torch.device,
    batch_size: int,
    variant_name: str,
) -> dict[str, SubsetStats]:
    results: dict[str, SubsetStats] = {}
    logit_scale = _logit_scale(model)

    for dataset_name, subset_name in SUBSET_ORDER:
        samples = datasets[dataset_name][subset_name]
        print(
            f"[benchmark] {variant_name}: evaluating {dataset_name}/{subset_name} "
            f"({len(samples)} images) ..."
        )
        image_features = _encode_images(
            model,
            processor,
            [sample.image for sample in samples],
            device,
            batch_size,
        )
        results[subset_name] = _evaluate_binary(
            samples,
            image_features,
            text_features,
            label_index,
            logit_scale,
        )
    return results


def format_comparison_table(
    model_reference: str,
    vanilla: dict[str, SubsetStats] | None,
    with_rn: dict[str, SubsetStats],
    rn_note: str,
) -> str:
    subset_w = max(len("Subset"), *(len(name) for _, name in SUBSET_ORDER))
    acc_w = 11
    margin_w = 14
    left_group_w = acc_w * 2 + 3
    right_group_w = margin_w * 2 + 3

    row_border = (
        "+-" + "-" * subset_w
        + "-+-" + "-" * acc_w
        + "-+-" + "-" * acc_w
        + "-+-" + "-" * margin_w
        + "-+-" + "-" * margin_w
        + "-+"
    )
    table_width = len(row_border)
    inner_width = table_width - 4
    top = "+" + "-" * (table_width - 2) + "+"
    title = f"Evaluated model: {_display_model_reference(model_reference)}"
    title_rows = [
        f"| {line.ljust(inner_width)} |"
        for line in (textwrap.wrap(title, width=inner_width) or [""])
    ]

    group_border = (
        "+-" + "-" * subset_w
        + "-+-" + "-" * left_group_w
        + "-+-" + "-" * right_group_w
        + "-+"
    )
    group_header = (
        f"| {'':{subset_w}} | "
        f"{'Accuracy'.center(left_group_w)} | "
        f"{'Mean logit margin'.center(right_group_w)} |"
    )
    column_header = (
        f"| {'Subset':<{subset_w}} | "
        f"{'Vanilla':>{acc_w}} | {'With RN':>{acc_w}} | "
        f"{'Vanilla':>{margin_w}} | {'With RN':>{margin_w}} |"
    )

    rows = []
    for _, subset_name in SUBSET_ORDER:
        vanilla_stats = vanilla[subset_name] if vanilla is not None else None
        rn_stats = with_rn[subset_name]
        vanilla_acc = f"{vanilla_stats.accuracy:.3f}" if vanilla_stats else "n/a"
        vanilla_margin = (
            f"{vanilla_stats.mean_logit_margin:+.3f}" if vanilla_stats else "n/a"
        )
        rows.append(
            f"| {subset_name:<{subset_w}} | "
            f"{vanilla_acc:>{acc_w}} | "
            f"{rn_stats.accuracy:>{acc_w}.3f} | "
            f"{vanilla_margin:>{margin_w}} | "
            f"{rn_stats.mean_logit_margin:>+{margin_w}.3f} |"
        )

    notes = [
        "Margin = logit(object) - logit(attack); positive favors the object.",
        rn_note,
    ]
    note_rows = []
    for note in notes:
        note_rows.extend(
            f"| {line.ljust(inner_width)} |"
            for line in (textwrap.wrap(note, width=inner_width) or [""])
        )

    return "\n".join(
        [
            top,
            *title_rows,
            group_border,
            group_header,
            column_header,
            row_border,
            *rows,
            row_border,
            *note_rows,
            top,
        ]
    )


def run_model(
    model_reference: str,
    rn_source: str,
    datasets: dict[str, dict[str, list[PairSample]]],
    labels: list[str],
    label_index: dict[str, int],
    device: torch.device,
    batch_size: int,
) -> tuple[dict[str, SubsetStats] | None, dict[str, SubsetStats], str]:
    print()
    print("=" * 78)
    print(f"[benchmark] Loading model: {model_reference}")
    model = CLIPModel.from_pretrained(model_reference).eval().to(device)
    print(f"[benchmark] Loading processor: {model_reference}")
    processor = CLIPProcessor.from_pretrained(model_reference)

    print(
        f"[benchmark] Encoding {len(labels)} unique object/attack labels with prompt: "
        f"{PROMPT!r}"
    )
    text_features = _encode_texts(model, processor, labels, device)

    status = read_null_status(model)
    if status is None:
        print("[benchmark] No active RN token detected. Running vanilla pass first.")
        vanilla = evaluate_variant(
            model,
            processor,
            datasets,
            text_features,
            label_index,
            device,
            batch_size,
            "vanilla",
        )
        print(f"[benchmark] Appending RN token from: {rn_source}")
        ensure_read_null_token(model, rn_source, debug=True)
        rn_note = f"RN was attached by rn_adapter.py from {rn_source}."
    else:
        print(f"[benchmark] RN already active ({status}); leaving it untouched.")
        print("[benchmark] Vanilla pass skipped because generic removal is unsafe.")
        vanilla = None
        rn_note = f"RN was already active in the loaded model ({status}); left unchanged."

    print("[benchmark] Running RN pass ...")
    with_rn = evaluate_variant(
        model,
        processor,
        datasets,
        text_features,
        label_index,
        device,
        batch_size,
        "with RN",
    )

    del text_features, processor, model
    if device.type == "cuda":
        torch.cuda.empty_cache()
    return vanilla, with_rn, rn_note


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Binary zero-shot SCAM/RTA benchmark across CLIP ViT-L/14 models"
    )
    parser.add_argument(
        "--models",
        nargs="+",
        default=list(DEFAULT_MODELS),
        help=(
            "Model repo IDs or local paths. Defaults: OpenAI ViT-L/14, "
            "zer0int CLIP-GmP, and the directory containing this benchmark."
        ),
    )
    parser.add_argument(
        "--rn-token",
        default=RN_ADAPTER_REPO_DIR,
        help="RN repo ID, local repo directory, or read_null_token.safetensors path",
    )
    parser.add_argument("--batch-size", type=int, default=32)
    parser.add_argument("--seed", type=int, default=DEFAULT_SEED)
    parser.add_argument("--device", default=None)
    args = parser.parse_args()

    print("[benchmark] Starting binary zero-shot typographic-attack benchmark.")
    configure_reproducibility(args.seed)

    device = torch.device(args.device or ("cuda" if torch.cuda.is_available() else "cpu"))
    print(f"[benchmark] Using device: {device}")
    print(f"[benchmark] RN source for models without RN: {args.rn_token}")
    print("[benchmark] Models:")
    for index, model_reference in enumerate(args.models, start=1):
        print(f"  {index}. {model_reference}")

    datasets = {
        "SCAM": load_scam_samples(),
        "RTA": load_rta_samples(),
    }
    all_samples = [
        sample
        for dataset_buckets in datasets.values()
        for samples in dataset_buckets.values()
        for sample in samples
    ]
    labels = sorted(
        {
            label
            for sample in all_samples
            for label in (sample.correct_label, sample.distractor_label)
        },
        key=str.casefold,
    )
    label_index = {label: index for index, label in enumerate(labels)}

    tables: list[str] = []
    for model_reference in args.models:
        vanilla, with_rn, rn_note = run_model(
            model_reference,
            args.rn_token,
            datasets,
            labels,
            label_index,
            device,
            args.batch_size,
        )
        tables.append(
            format_comparison_table(
                model_reference, vanilla, with_rn, rn_note
            )
        )

    print()
    print("[benchmark] Evaluation complete. Results:")
    print()
    print("\n\n".join(tables))


if __name__ == "__main__":
    main()

🎯 Evaluation

Alias	Model
OAI	openai/clip-vit-large-patch14
GmP	zer0int/CLIP-GmP-ViT-L-14
RNb	zer0int/CLIP-ViT-L-14-Universal-VPT-ReadNull-Token

(--) = RN removed / vanilla image encoder
(RN) = RN token inserted

Overview: Typographic Attack Benchmark, without (light) and with (solid color) RN Token.
From left or right: OAI, GmP, RNb (Read-Null base; this repo, model.safetensors)

Typographic attacks — RTA-100 + SCAM

Binary accuracy

Subset	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
NoSCAM	0.990	0.986	0.988	0.989	0.989	0.989
SCAM	0.416	0.577	0.640	0.809	0.769	0.893
SynthSCAM	0.315	0.601	0.607	0.847	0.748	0.918
NoRTA	0.988	0.989	0.992	0.991	0.992	0.993
RTA	0.440	0.630	0.614	0.788	0.729	0.871
SynthRTA	0.402	0.641	0.611	0.804	0.744	0.910

Mean logit margin

Positive values favor the correct object label over the typographic attack label.

Subset	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
NoSCAM	+9.580	+9.233	+18.434	+18.481	+16.638	+16.443
SCAM	-0.530	+1.028	+2.842	+7.012	+4.929	+8.080
SynthSCAM	-1.817	+0.852	+1.484	+7.842	+3.836	+9.107
NoRTA	+9.242	+8.941	+17.537	+17.463	+15.556	+15.282
RTA	-0.513	+1.248	+1.970	+6.211	+3.835	+7.240
SynthRTA	-1.122	+1.061	+1.332	+6.526	+3.426	+8.300

---

ImageNet-1k linear probe

Metric	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
Top-1	—	71.320%	—	72.310%	—	72.910%
Top-5	—	93.560%	—	94.280%	—	94.590%

---

ObjectNet MVT zero-shot classification

Accuracy

	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
ObjectNet MVT	0.860	0.856	0.881	0.863	0.879	0.866

Mean margin

	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
ObjectNet MVT	+0.036	+0.035	+0.074	+0.070	+0.065	+0.060

---

MSCOCO retrieval

Higher recall is better; lower MeanR is better.

Metric	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
I2T R@1	0.571	0.560	0.690	0.680	0.693	0.685
I2T R@5	0.800	0.798	0.886	0.880	0.881	0.879
I2T R@10	0.872	0.876	0.935	0.931	0.932	0.930
I2T MeanR ↓	6.570	6.480	3.400	3.590	3.520	3.630
T2I R@1	0.354	0.368	0.517	0.511	0.508	0.502
T2I R@5	0.604	0.619	0.769	0.761	0.759	0.754
T2I R@10	0.710	0.722	0.850	0.844	0.840	0.836
T2I MeanR ↓	21.460	20.110	10.250	10.400	10.610	10.710

---

SugarCrepe

Accuracy

Subset	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
add_obj	0.786	0.786	0.928	0.927	0.921	0.920
add_att	0.720	0.743	0.837	0.845	0.814	0.817
replace_obj	0.941	0.943	0.969	0.970	0.969	0.970
replace_att	0.796	0.787	0.868	0.872	0.860	0.863
replace_rel	0.654	0.661	0.770	0.768	0.766	0.756
swap_obj	0.608	0.588	0.727	0.714	0.714	0.694
swap_att	0.635	0.622	0.697	0.710	0.704	0.698

Mean margin

Subset	OAI (--)	OAI (RN)	GmP (--)	GmP (RN)	RNb (--)	RNb (RN)
add_obj	+1.389	+1.300	+3.752	+3.641	+3.255	+3.172
add_att	+0.848	+0.905	+1.785	+1.753	+1.439	+1.411
replace_obj	+5.063	+4.850	+10.442	+10.218	+9.571	+9.328
replace_att	+1.830	+1.731	+3.606	+3.504	+3.253	+3.187
replace_rel	+0.948	+0.902	+1.923	+1.877	+1.624	+1.592
swap_obj	+0.365	+0.333	+1.215	+1.108	+1.034	+0.952
swap_att	+0.503	+0.447	+1.289	+1.205	+1.138	+1.088

---

Love ❤️ this CLIP?
ᐅ Buy me a coffee on Ko-Fi ☕

Or click here for address to send 🪙₿ BTC

3PscBrWYvrutXedLmvpcnQbE12Py8qLqMK