Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations (2022)

Abstract: We examine the effects of contrastive visual semantic pretraining by comparing the geometry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal image classifier which adapts the GPT-2 architecture to encode image captions. We find that contrastive visual semantic pretraining significantly mitigates the anisotropy found in contextualized word embeddings from GPT-2, such that the intra-layer self-similarity (mean pairwise cosine similarity) of CLIP word embeddings is under .25 in all layers, compared to greater than .95 in the top layer of GPT-2. CLIP word embeddings outperform GPT-2 on word-level semantic intrinsic evaluation tasks, and achieve a new corpus-based state of the art for the RG65 evaluation, at .88. CLIP also forms fine-grained semantic representations of sentences, and obtains Spearman’s \rho = .73 on the SemEval-2017 Semantic Textual Similarity Benchmark with no fine-tuning, compared to no greater than \rho = .45 in any layer of GPT-2. Finally, intra-layer self-similarity of CLIP sentence embeddings decreases as the layer index increases, finishing at .25 in the top layer, while the self-similarity of GPT-2 sentence embeddings formed using the EOS token increases layer-over-layer and never falls below .97. Our results indicate that high anisotropy is not an inevitable consequence of contextualization, and that visual semantic pretraining is beneficial not only for ordering visual representations, but also for encoding useful semantic representations of language, both on the word level and the sentence level.

## introduction large-scale "natural language supervision" using image captions collected from the internet has enabled the first "zero-shot" artificial intelligence (ai) image classifiers, which allow users to create their own image classes using natural language, yet outperform supervised models on common language-and-image tasks @xcite . the image encoders of such models have been shown to form "multimodal" representations in the upper layers, such that the same neurons fire for photographic, symbolic, and textual depictions of a concept @xcite . research on these state of the art "visual semantic" (joint language-and-image) models has focused primarily on their benefits for encoding semantically legible representations of images. in this paper, we seek to answer a straightforward but as yet unexplored question: what benefits does contrastive visual semantic pretraining have for representations of natural language? the clip ("contrastive language image pretraining") image classification model introduced by @xcite provides a unique opportunity to observe the effects of visual semantic pretraining on a contextualizing language model. while most other visual semantic architectures combine language and image features in the inner layers of the model @xcite , clip separates the language model from the vision model until the end of the encoding process, at which point it projects a representation formed by each model into a joint language-image embedding space @xcite . clip is trained to maximize the cosine similarity of a projected image with its projected natural language caption, while minimizing the cosine similarity of the projected caption with all of the other images in the batch @xcite , a training objective known as "contrastive learning" or "contrastive representation distillation" @xcite . the separation of the language model from the vision model prior to projection allows us to consider the two models independently of each other, such that we can study representations of natural language trained for a visual semantic objective, rather than representations which combine language and image features in the inner layers of the model. moreover, because clip encodes natural language using gpt-2, a "causal" language 0 1 2 3 4 5 6 7 8 9 10 11 12 0 0.2 0.4 0.6 0.8 1 language model layer self-similarity cwe self-similarity by language model layer gpt-2 clip figure 1 : clip cwes are much less self-similar than gpt-2 cwes, despite being trained with the same architecture, suggesting that pretraining objective leads to significant differences in contextualized representations which are not the result of the contextualization process itself, nor of the model architecture. we observe differences between representations formed by gpt-2 and the clip language model ("lm") both on the word level and on the sentence level. we outline our contributions: we make our code and data available at https://github.com/wolferobert3/clip_ contrastive_acl_2022. ## related work we review prior work on visual semantic ai, on the geometry and semantic properties of representations formed by language models, and on semantic intrinsic evaluation tasks. ## data for comparison of our results on cwe anisotropy with the prior work of @xcite , we encode the text of the semeval semantic textual similarity tasks from 2012 through 2016 @xcite @xcite , who used these datasets because they include instances of the same words used in different contexts and reflecting different word senses. we discard sentences too long to fit in the 77-token context window of the clip lm, which still leaves us with over 36,000 sentences. ## approach and experiments we outline our experiments, and discuss our approach for extracting both cwes and sentence embeddings, and for computing self-similarity. ## results clip cwes are less anisotropic than gpt-2 embeddings, and clip outperforms gpt-2 on wordlevel and sentence-level semantic evaluations. ## discussion our findings are straightforward, but it is not obvious that they should occur. the training objective of clip is not to produce high-quality cwes, or even sentence embeddings. indeed, @xcite spend little time discussing the clip language model, noting that they did not see significant performance improvements by scaling up the size of the model. however, in creating the first broadly accurate zero-shot image classifier, @xcite have also created a zero-shot sentence encoder which substantially outperforms the version of its underlying architecture trained on language modeling. moreover, without the need for computationally expensive pooling methodologies, and despite having less than half the parameter count of gpt-2 small, the clip lm produces cwes which match or exceed the best performance ever realized with a corpus-based approach on two intrinsic evaluation tasks, and outperform embeddings formed from 100, 000 encoded contexts in gpt-2 medium @xcite . clip embeddings show that the high anisotropy observed by @xcite is not the inevitable result of contextualization, nor even of a specific language modeling architecture, but is connected to the pretraining objective of the model. when trained on a contrastive visual semantic objective, cwes formed by clip have much lower selfsimilarity at every layer of the model in comparison with gpt-2. this is remarkable because clip does not actually project cwes into the joint languageimage space. while we might expect clip sentence embeddings, which are projected into the language-image space, to have different properties from the cwes formed by gpt-2, it does not necessarily also follow that the cwes formed by clip would also be so different from those in gpt-2. indeed, we still observe the increased self-similarity in the top layer reported by @xcite , and the loss of semantic information related to the input token in the upper layers, as reported by @xcite . however, these effects are much less pronounced in clip than they are in gpt-2, indicating that the contrastive visual semantic objective of the model has regularizing effects that shape more than just the projected sentence embedding. our findings suggest that language models trained on visual semantic objectives are likely to privilege the encoding of semantic information, which is essential to matching a caption to an image. the more isotropic representations we observe reflect the objective of the model, which requires differentiating fine-grained semantic information. that models trained on visual semantic objectives would form embeddings to reflect the semantics of a word or sentence more than would a causal language model makes intuitive sense. through the lens of the training objective, it is more problematic for a causal language model to predict a syntactically invalid continuation of a sentence, such as an incorrect part of speech, than to predict a somewhat unexpected but still syntactically valid continuation of a sentence. when a language model is trained to encode and associate the correct text caption with a matching image, however, the semantic content of the text becomes at least as important as its syntactic properties. ## conclusion we find that contrastive visual semantic pretraining produces isotropic cwes which outperform a language model based on the same architecture on semantic evaluations on both the word level and the sentence level. our findings indicate that incorporating visual semantic objectives with language models may be useful both to regularize the anisotropy in cwes and to improve the semantic quality of both word and sentence representations.