Data or Language Supervision: What Makes CLIP Better than DINO ? (2025)

Abstract: CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP’s language supervision or its much larger training data. To disentangle these factors, we pre-train CLIP and DINO under controlled settings—using the same architecture, dataset, and training configuration—achieving similar ImageNet accuracy. Embedding analysis shows that CLIP captures high-level semantics (e.g., object categories, text), while DINO is more responsive to low-level features like colors and styles. When integrated into VLMs and evaluated on 20 VQA benchmarks, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones. Variants of language supervision (e.g., sigmoid loss, pre-trained language encoders) yield limited gains. Our findings provide scientific insights into vision encoder design and its impact on VLM performance.

## introduction vision-language models, such as @xcite and claude @xcite , have demonstrated transformative capabilities in interpreting and reasoning over visual inputs. these models typically consist of a vision encoder, a language model, and a connector module bridging the two @xcite . the vision encoder-serving as the "eyes" of the system-plays a critical role in transmitting visual information to the language model, which acts as the "brain" that interprets it. recent studies have shown that clip @xcite , particularly its variant siglip @xcite , has emerged as the most effective vision encoder for building vlms, significantly outperforming dino-based counterparts across various domains @xcite data × epochs v100 gpu hours (2) clip and its variant siglip are trained on datasets that are up to 100 times larger than those used for dino. these differences make it difficult to disentangle whether clip's superior performance in visionlanguage models stems from its training objective or the scale of its training data. to isolate these factors, we conduct a controlled study by training clip and dino vision encoders under identical conditions: the same architecture (vit-b/16), dataset (a 10m-image subset of datausing these controlled encoders, we first investigate how language supervision alters the embedding space. we identify and analyze image pairs where clip and dino produce significantly different similarity scores. our analysis reveals that clip is more sensitive to high-level visual semantics-such as object type and embedded text-while dino is more responsive to low-level visual attributes like colors and styles. we then integrate these controlled encoders into the llava @xcite and train the resulting vlms under identical settings. evaluated on 20 vqa benchmarks, llava-clip significantly outperforms llava-dino on text-intensive tasks (e.g., questions involving tables or charts), achieving a 7.5% performance gain. llava-dino performs slightly better on some visually grounded tasks but matches llava-clip on most others. to further probe the effect of language supervision, we explore two additional questions: (1) do different supervision objectives, such as clip vs. siglip, lead to performance differences? (2) does using a pre-trained language encoder during clip training yield a stronger vision encoder? in both cases, we find the answer to be no. in summary, our study examines how vision encoders influence the performance of visionlanguage models. through carefully controlled experiments, we uncover the representational differences induced by language supervision and their downstream effects, thereby offering the community deeper scientific insights into designing visioncentric vision-language systems. ## training controlled clip and dino in this section, we describe how we train clip and dino under controlled settings, ensuring that the only difference lies in the supervision signal. we then evaluate their performance on various image classification benchmarks. ## embedding analysis to gain deeper insight into how language supervision shapes the embedding space, we conduct a fine-grained embedding analysis comparing clip and dino. unlike coarse-grained metrics like classification accuracy, this analysis reveals how each model organizes visual information. method. similar to tong et al. @xcite , we analyze pairs of images in datacomp-10m where clip and dino produce highly divergent similarity scores, revealing systematic differences in representation. specifically, we identify two types of image pairs: these selected pairs help isolate cases where the two models disagree in their embeddings. results. figure 2 illustrates representative examples for our analysis. clip shows strong alignment with high-level semantic features such as object identity and textual content. it consistently groups images by object type or embedded texts, even across variations in visual style or context-suggesting that language supervision enhances semantic abstraction. in contrast, dino is more sensitive to low-level visual cues like color schemes, and is more invariant to orientation change. we provide a quantitative validation for these observations in appendix b. these findings highlight that clip learns embeddings that are more semantically meaningful, while dino emphasizes visual similarity, likely due to its selfsupervised objective. ## vlm analysis after training the controlled clip and dino encoders, we incorporate them into the llava-1.5 framework to investigate how vision encoder choice impacts the performance of vlms. experimental setup. we use llava-1.5 with its vision encoder replaced by either controlled clip or dino. training consists of pretraining followed by visual instruction tuning. during training, we save checkpoints every 500 steps and evaluate each on vmcbench @xcite )-a unified multiple-choice visual question answering benchmark composed of 20 datasets-to simplify evaluation. since test set labels are not publicly available, we select the best checkpoint based on validation performance and report validation results. all training configurations are kept identical for both clip and dino versions. ## exploring better language supervision given that language supervision (1) improves finegrained image classification, (2) encourages highlevel semantic alignment, and (3) enhances ocr task performance in vlms, we further explore whether alternative forms of language supervision can yield stronger vision encoders. ## conclusion this work conducts a controlled study to disentangle the effects of language supervision and data scale on vision encoder performance in vlms, offering insights into vision encoder design and its role in effective vlms.