TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision
Authors: Emran Abdu
DOI: 10.5281/zenodo.21180996
Code: GitHub
License: Apache 2.0
Abstract
Vision foundation models like DINOv2 produce powerful representations, but training them costs millions of dollars in GPU compute. We introduce TinySSL, a 2.8M-parameter framework that distills frozen DINOv2-S/14 features into a compact CNN-transformer hybrid. A composite loss combines masked image modeling with JEPA alignment, cosine feature matching, and KoLeo uniformity regularization, removing the need for negative pairs, momentum encoders, or large batches. A progressive augmentation curriculum stabilizes training on commodity hardware. Across four domain benchmarks (Flowers102, Oxford Pets, EuroSAT, BreastMNIST), TinySSL retains over 97% of DINOv2-S/14 linear-probe accuracy with a 7x parameter reduction and trains in under 30 minutes on a single CPU.
Citation
ibtex @article{abdu2026tinyssl, title={TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision}, author={Emran Abdu}, year={2026}, doi={10.5281/zenodo.21180996} }