On the Language Encoder of Contrastive Cross-modal Models (2024)

Abstract: Contrastive cross-modal models such as CLIP and CLAP aid various vision-language (VL) and audio-language (AL) tasks. However, there has been limited investigation of and improvement in their language encoder – the central component of encoding natural language descriptions of image/audio into vector representations. We extensively evaluate how unsupervised and supervised sentence embedding training affect language encoder quality and cross-modal task performance. In VL pretraining, we found that sentence embedding training enhances language encoder quality and aids in cross-modal tasks, improving contrastive VL models such as CyCLIP. Sentence embedding training benefits AL tasks when the amount of training data is large. We analyze the representation spaces to understand the strengths of sentence embedding training, and find that it improves text-space uniformity, at the cost of decreased cross-modal alignment.

## introduction significant progresses have been made in pretraining large-scale cross-modal models, such as clip @xcite and align @xcite , for various vision-language (vl) applications such as retrieval and zero-shot image classification. these models are often pretrained with large amounts of data, e.g., openai leverages «400m caption-image pairs to train clip while laion-ai scaled up this number to 5b @xcite . such a large amount of multi-modal pretraining data contains text captions at the same scale as the pretraining corpora of large language models (llms) such as bert, which is pretrained on 3.3b words @xcite . the success of vl pretraining encourages research on contrastive learning models for other modalities like audio. pretrained audio-language (al) models such as audioclip @xcite and clap @xcite show promising results on al retrieval and zero-shot audio classification tasks. it is clear that the language encoder in crossmodal contrastive models plays a central role when scaling-up pretraining of a specific modality and/or the amount of modalities. therefore, analyzing and improving the language encoder become increasingly crucial. clip's language encoder (clip lm) -a decoder-only language model similar to gpt-2 @xcite -has been investigated. @xcite showed that the clip lm outperforms bert @xcite in clustering entities with prompting. @xcite probed the clip lm, showing its word representations are less anisotropic @xcite , i.e., more uniformly distributed with respect to direction, than gpt-2. complementary to these research on clip lm, we focus on pretraining. clip-like models are often pretrained with crossmodal contrastive learning. we measure -during pretraining -the effectiveness of systematically modeling the captions with sentence embedding training @xcite , which is a natural fit to the captions. we pretrain clip and one of its new variants cy-clip @xcite with sentence embedding training, as well as the conventional cross-modal contrastive learning. in addition to clip's crossmodal contrastive learning objective, cyclip explicitly optimizes for geometry consistency between the text and image representation spaces, making it a suitable model for validating the effectiveness of nlp methods. we evaluate pretrained models on an array of tasks involving one or two modalities such as senteval, zero-shot vl retrieval, and image classification. we find that unsupervised sentence embedding training improves the language encoder quality and vl tasks. supervised sentence embedding training improves language en-coder quality, but the benefit does not necessarily transfer to vl tasks. we analyze the learned representation spaces and find that sentence embedding training improves text-space uniformity @xcite and reduces anisotropy @xcite . we also investigate al contrastive models such as clap @xcite . in contrast to vl pretraining, al pretraining suffers from data scarcity and often leverages pretrained llms and audio encoders. we determine the effectiveness of sentence embedding training in both scenarios: continued pretraining with llms and audio encoders, and pretraining from scratch. we find that sentence embedding training improves al tasks when the amount of data is large, while the benefits become less noticeable on small datasets. to the best of our knowledge, this is the first study on investigating and trying to improve the language encoder of al contrastive learning, and we expect our results will encourage more research in this direction. in summary, our contributions are as follows: (i) we extensively evaluate how unsupervised and supervised sentence embedding trainings affect vl and al contrastive pretraining. experimental results indicate improved vl performance. al tasks see improvements when the amount of training data is large, while the benefits become less noticeable on small datasets. (ii) we show that unsupervised sentence embedding training improves the language encoder of cyclip @xcite , hence improves performance of cross-modal tasks. (iii) we conduct a comprehensive analysis on the alignment and uniformity of learned representation spaces following @xcite , and show that sentence embedding training improves uniformity of the text representation space, but at the cost of decreased cross-modal alignment. contrastive audio-language pretraining models have also been recently proposed. @xcite extended clip to audio tasks by adding an extra module and continued training on audio datasets. similar distillation methods such as wav2clip @xcite , have also been proposed. @xcite and @xcite independently proposed clap, in which a language encoder and an audio encoder are jointly trained on al datasets, which resembles clip. we focused on the language encoder in the al models, and demonstrated the impact of sentence embedding training. to the best of our knowledge, this is the first step in this direction. sentence embedding is an extensively investigated nlp topic. methods ranging from bagof-word averaging non-contextualized embeddings @xcite to training lstms @xcite , e.g., skipthought @xcite and infersent @xcite , have been proposed to effectively compose individual tokens to meaningful sentence representations. methods that leverage post-hoc transforming @xcite or finetuning the pretrained bert in supervised @xcite or unsupervised scenarios @xcite are also introduced. @xcite show that grounding sentence embedding learning to images improves semantic textual similarity tasks. we present a focused investigation on the lm in clip/cyclip. we pretrained from scratch an lm and resnet-50 @xcite with cross-modal contrastive learn-ing, as well as unsupervised or supervised sentence embedding training with image captions. verifying the effectiveness of sentence embeddings -a critical component for retrieval and clustering @xcite @xcite @xcite -is of great importance because retrieval has been one of the main applications of clip-like models. ## method cross-modal contrastive learning plays a key role in training models such as clip/clap. we take the image modality as an example for introducing this method. consider a caption-image dataset tpi i , t i qu n i"1 that includes n caption-image pairs, and denote i e and t e as the output representations from an image and language encoder, respectively. the cross-modal contrastive loss @xcite is defined as intuitively, decreasing the cross-modal consistency loss l c-cyclic makes the cross-modal similarity matrix more symmetric, as shown in figure 1. note that solely optimizing l contra. is expected to symmetrize the cross-modal similarity matrix because the similarity of non-diagonal pairs are trained to be zero. @xcite showed that this scenario does not occur in practice and explicitly optimizing l c-cyclic is beneficial. optimizing l i-cyclic , however, reduces the inconsistency between the overall geometry of the text and image spaces. computing l i-cyclic requires calculating the similarity between two text captions, i.e., xt e k , t e j y. it is thus reasonable to hypothesize that accurately computing caption similarities is beneficial for optimizing l i-cyclic . we use unsupervised and supervised sentence embedding training methods and test this hypothesis ( §4). we use the widely used simcse @xcite ) method for unsupervise learning caption representations. simcse also uses contrastive learning: a caption is input to a language encoder twice to obtain two vectors t e and t e `. with dropout @xcite ) enabled, t e and t e `are generally different. these paired vectors serve as the positive training pairs for contrastive learning, while mismatched captions form negative pairs. we denote the unsupervised simcse loss as another direction of sentence embedding training is supervised training @xcite on natural language inference (nli) datasets, e.g., @xcite . we denote the objective as l n and follow @xcite in using entailment pairs as t e and t e as a result, the overall training objective considered is: ## conclusion we extensively investigate the effectiveness of sentence embedding training for pretraining contrastive vision-language and audio-language models. we show that it improves vision-language pretraining, resulting in a better cyclip. sentence embedding training also improves audio-language pretraining on large datasets, while the benefits diminish on small datasets. we conduct comprehensive analyses and show that sentence embedding training increases text space uniformity, but with a cost of reduced cross-modal alignment.