MCSE : M ultimodal Contrastive Learning of Sentence Embeddings (2022)

Abstract: Learning semantically meaningful sentence embeddings is an open problem in natural language processing. In this work, we propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. Through experiments on a variety of semantic textual similarity tasks, we demonstrate that our approach consistently improves the performance across various datasets and pre-trained encoders. In particular, combining a small amount of multimodal data with a large text-only corpus, we improve the state-of-the-art average Spearman’s correlation by 1.7%. By analyzing the properties of the textual embedding space, we show that our model excels in aligning semantically similar sentences, providing an explanation for its improved performance.

## introduction sentence embedding learning, i.e., encoding sentences into fixed-length vectors that faithfully reflect the semantic relatedness among sentences, is a fundamental challenge in natural language processing (nlp). despite the tremendous success of pre-trained language models (plms), such as bert @xcite and roberta @xcite , it has been shown that the off-theshelf sentence embeddings of plms without finetuning are even inferior to averaging glove embeddings @xcite in terms of semantic similarity measure @xcite . hence, recent research @xcite @xcite focuses on adjusting the original sentence embeddings derived from plms in an unsupervised manner. in particular, there has been growing interest in adopting contrastive learning objectives to achieve this goal @xcite @xcite . although purely text-based models have led to impressive progress, it remains an open question to what extent they capture the deeper notion of sentence meaning beyond the statistical distribution of texts, which lies outside of the text and is grounded in the real-world @xcite . as a central part of the human perceptual experience, vision has been shown to be effective in grounding language models and improving performance on various nlp tasks @xcite @xcite . we hypothesize that using vision as supplementary semantic information can further promote sentence representation learning. in this work, we propose mcse, an approach for multimodal contrastive learning of sentence embeddings. to exploit both visual and textual information, we adopt the state-of-the-art contrastive sentence embedding framework simcse @xcite and extend it with a multimodal contrastive objective. in addition to the textual objective in simcse that maximizes agreement between positive sentence pairs, the multimodal objective maximizes agreement between sentences and corresponding images in a shared space. we conduct extensive experiments on standard semantic textual similarity (sts) benchmarks and show the effectiveness of mcse across various datasets and pre-trained encoders. we find that, using a small amount of multimodal data in addition to a textonly corpus yields significant improvements on sts tasks. by analyzing the alignment and uniformity properties of the embedding space @xcite , we show that mcse better aligns the semantically similar sentences while maintaining uniformity, providing an explanation for its superior performance. ## method to exploit both visual and textual information, we adopt simcse @xcite as the textual baseline and extend it with a multimodal contrastive learning objective. ## limitations despite showing performance improvements on sts benchmarks, mcse has its limitations as well. we take caption datasets as the source of multimodal information, while these datasets are collected and curated with non-negligible human efforts. it will have great practical value if we can properly leverage noisy image-sentence pairs or even get rid of the explicit alignments between imrelease the code for calculating these two losses, the absolute values we obtained might be different from theirs. we make sure our calculation across different models is consistent. ## conclusion in this paper, we propose mcse, a novel approach for sentence embedding learning that applies a multimodal contrastive objective to align sentences and corresponding images in a grounded space. experiments show that mcse consistently improves the performance on sts tasks. we also highlight the superiority of our method by analyzing the alignment and uniformity properties of the embedding space. the multimodal objective is generic and can be potentially incorporated into other sentence embedding methods to boost their performance. eloi zablocki, benjamin piwowarski, laure soulier, and patrick gallinari. 2018. learning multi-modal word representation grounded in visual context. in proceedings of the aaai conference on artificial intelligence. yan zhang, ruidan he, zuozhu liu, kwan hui lim, and lidong bing. 2020. an unsupervised sentence embedding method by mutual information maximization. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 1601-1610. zhuosheng zhang, kehai chen, rui wang, masao utiyama, eiichiro sumita, zuchao li, and hai zhao. 2019. neural machine translation with universal visual representation. in international conference on learning representations (iclr). yanpeng zhao and ivan titov. 2020. visually grounded compound pcfgs. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 4369-4379.