hasanfaesal/vit-tiny-patch16-224-dinov2-distill

ViT-Tiny/16-224 distilled from DINOv2 (DataComp)

A 5.7M-parameter ViT-Tiny/16 trained from scratch (no ImageNet supervision, no pretrained initialisation) by distilling a frozen DINOv2 ViT-S/14 (dinov2_vits14_reg) teacher on a 5.9M-image subset of DataComp-12M.

What makes it work is exact patch-grid matching: the student runs at 224×224 with patch size 16 → a 14×14 = 196-token grid, and the teacher is fed the same view resized to 196×196 with patch size 14 → also a 14×14 = 196-token grid. Every student patch token therefore has a 1-to-1 spatial counterpart in the teacher, so dense patch-level regression needs no interpolation, no pooling and no attention matching.

	Run 1	Run 2 (default)
Images (DataComp-12M subset)	1.22M	5.94M
Epochs	40	30
CIFAR-100 k-NN (k=20)	75.50%	76.25%
CIFAR-100 linear probe	65.72%	66.97%
Pascal VOC patch mIoU	56.97%	56.41%
File	run1_model.safetensors	model.safetensors

Run 2 is the default export and wins on every classification benchmark. Run 1 is marginally better on the dense VOC patch probe (+0.56 mIoU), so both are published.

---

Method

Teacher — facebookresearch/dinov2 → dinov2_vits14_reg (ViT-Small/14, 21.8M params, 384-dim, 4 register tokens), frozen, bf16, eval mode, requires_grad=False.

Student — timm vit_tiny_patch16_224 (192-dim, 12 blocks, 3 heads, 5.7M params), random init, bf16 autocast, torch.compile(mode="reduce-overhead").

Views — 2 global views per image at 224×224: random resized crop (scale 0.4–1.0, bicubic), horizontal flip, colour jitter (0.4 / 0.4 / 0.2, p=0.8), Gaussian blur (p=0.5), solarisation (p=0.2). Each view is resized bicubic to 196×196 for the teacher.

Projection heads

cls_head   = nn.Sequential(nn.Linear(192, 1024), nn.GELU(), nn.Linear(1024, 384))
patch_head = nn.Linear(192, 384)

Loss — direct cosine feature regression against the frozen teacher:

$$\mathcal{L}=\underset{{\mathcal{L}}_{cls}}{\underbrace{1-\cos (\mathrm{c}\mathrm{l}\mathrm{s}\mathrm{\_}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}(S_{cls}),T_{cls})}}+\lambda \cdot \underset{{\mathcal{L}}_{patch}}{\underbrace{\frac{1}{196}\sum _{i=1}^{196}(1-\cos (\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\mathrm{\_}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}(S^{(i)}),T^{(i)}\left)\right)}},\lambda =1.0\backslash mathcal\{L\}\; \backslash ;=\backslash ;\; \backslash underbrace\{1\; -\; \backslash cos\backslash big(\backslash mathrm\{cls\backslash \_head\}(S\_\{cls\}),\backslash ;\; T\_\{cls\}\backslash big)\}\_\{\backslash mathcal\{L\}\_\{cls\}\}\; \backslash ;+\backslash ;\; \backslash lambda\; \backslash cdot\; \backslash underbrace\{\backslash frac\{1\}\{196\}\backslash sum\_\{i=1\}^\{196\}\backslash Big(1\; -\; \backslash cos\backslash big(\backslash mathrm\{patch\backslash \_head\}(S^\{(i)\}),\backslash ;\; T^\{(i)\}\backslash big)\backslash Big)\}\_\{\backslash mathcal\{L\}\_\{patch\}\},\; \backslash qquad\; \backslash lambda\; =\; 1.0$$

The teacher is frozen, so there is no collapse risk and no centering / sharpening is needed — a plain cosine regression is enough.

Optimisation — AdamW, lr 1e-3, weight decay 0.05, linear warmup then cosine annealing, gradient clipping at 3.0, batch 256 images (512 views) per step, bf16, on 1× NVIDIA H100 80GB SXM5 (~12.4 h wall-clock for Run 2).

---

Files

File	Contents
model.safetensors	Run 2 backbone, native timm key names (canonical)
run1_model.safetensors	Run 1 backbone, native timm key names
distill_run2_full.safetensors	Run 2 full student module — backbone.* + cls_head.* + patch_head.* (158 tensors)
distill_run1_full.safetensors	Run 1 full student module (158 tensors)
config.json	timm-native config (architecture, input size, normalisation)
run_metrics.json	Full training config + all benchmark numbers
checkpoint_metadata.json	Provenance extracted from the raw training checkpoints

The backbone exports hold 150 tensors in native timm naming (cls_token, pos_embed, patch_embed.proj.*, blocks.0..11.*, norm.*). The ImageNet classification head is not included — it was never trained by distillation, so the model is published as a pure feature extractor (num_classes: 0). Everything is published as safetensors only.

---

Usage

import torch, timm
from timm.data import resolve_data_config, create_transform

model = timm.create_model(
    "hf-hub:hasanfaesal/vit-tiny-patch16-224-dinov2-distill",
    pretrained=True,
).eval()

# the student was trained with ImageNet normalisation, not timm's 0.5/0.5 default
cfg = resolve_data_config({}, model=model)     # mean/std are read from config.json
transform = create_transform(**cfg)

x = torch.randn(2, 3, 224, 224)
with torch.no_grad():
    feats = model.forward_features(x)   # [2, 197, 192]  (CLS + 196 patch tokens)
    cls    = feats[:, 0]                # [2, 192]
    patches = feats[:, 1:]              # [2, 196, 192]

The reported k-NN numbers were produced by resizing the full image to 224×224 (bicubic) with no cropping.

Run 1, and the full distillation module (backbone + projection heads, useful to continue distillation or to map features into the teacher's 384-dim space):

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
import torch.nn as nn, timm

REPO = "hasanfaesal/vit-tiny-patch16-224-dinov2-distill"

# Run 1 backbone
model = timm.create_model("vit_tiny_patch16_224", pretrained=False, num_classes=0)
model.load_state_dict(load_file(hf_hub_download(REPO, "run1_model.safetensors")), strict=True)

# full student module -- key names match the reference implementation
class Student(nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = timm.create_model("vit_tiny_patch16_224", pretrained=False)
        self.cls_head = nn.Sequential(nn.Linear(192, 1024), nn.GELU(), nn.Linear(1024, 384))
        self.patch_head = nn.Linear(192, 384)

    def forward(self, x):
        f = self.backbone.forward_features(x)
        return self.cls_head(f[:, 0]), self.patch_head(f[:, 1:])

student = Student()
student.load_state_dict(
    load_file(hf_hub_download(REPO, "distill_run2_full.safetensors")), strict=True
)

---

Evaluation

Protocol: backbone CLS features, L2-normalised; k-NN with k=20, cosine similarity and a temperature-0.07 weighted vote; the linear probe trains a linear classifier for 15 epochs on frozen features. The teacher is evaluated identically (input resized to 196×196).

Benchmark	Metric	Run 1	Run 2	Teacher dinov2_vits14_reg
CIFAR-100	k-NN top-1 (k=20)	75.50%	76.25%	84.12%
CIFAR-100	Linear probe top-1	65.72%	66.97%	79.48%
Imagenette	k-NN top-1 (k=20)	97.58%	97.83%	99.41%
Imagenette	Linear probe top-1	95.69%	95.90%	98.55%
Pascal VOC 2012	Patch mIoU	56.97%	56.41%	68.23%

The patch-token regression transfers >82% of the teacher's segmentation quality with a quarter of the parameters.

Emergent object segmentation — last-layer [CLS] self-attention of the distilled ViT-Tiny (Run 2), with a single attention head per image (chosen by eye). The model never saw a segmentation or detection label: the objects light up purely as a by-product of the patch-level feature regression.

The full training configuration, every per-run metric and the feature-geometry numbers are in run_metrics.json.

---

Intended use and limitations

• Intended use — a small, strong general-purpose visual feature extractor: retrieval, linear probing, dense prediction, clustering, or a starting point for fine-tuning. It has no classification head; attach your own.
• The numbers come from a single seed per run and from CIFAR-100 / Imagenette / VOC — they are not full ImageNet-1k numbers, and a linear probe on ImageNet-1k would be the natural next benchmark.
• Distillation of DINOv2 reproduces the teacher's features, not its licence terms. The teacher weights are Apache-2.0; the DataComp-12M images carry their own copyrights (the metadata is CC-BY-4.0, the images are not). DINOv2 is known to be strong but not free of bias — treat downstream metrics on non-western datasets with care.

---

Citation

@software{hasanfaesal_vit_tiny_patch16_224_dinov2_distill,
  author    = {Hasan Faesal},
  title     = {ViT-Tiny/16-224 distilled from DINOv2 on DataComp-12M},
  year      = {2026},
  url       = {https://huggingface.co/hasanfaesal/vit-tiny-patch16-224-dinov2-distill}
}

@article{oquab2024dinov2,
  title   = {DINOv2: Learning Robust Visual Features without Supervision},
  author  = {Oquab, Maxime and Darcet, Timoth{\'e} and Moutakanni, Huy and Vo, Huy V. and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaa and Joulin, Armand and Jegou, Hervé and Mairal, J{\'e}r{\'e} and Labatut, Patrick and Joulin, Armand and Bojanowski, Piotr},
  journal = {arXiv preprint arXiv:2304.07193},
  year    = {2023}
}

@article{li2024datacomp,
  title   = {DataComp: In search of the next generation of multimodal datasets},
  author  = {Li, Xiaohan and others},
  journal = {arXiv preprint arXiv:2304.14108},
  year    = {2023}
}