I nfo CSE : Information-aggregated Contrastive Learning of Sentence Embeddings (2022)

Abstract: Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the original sentence fragments. Therefore, this paper proposes an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE.InfoCSE forces the representation of [CLS] positions to aggregate denser sentence information by introducing an additional Masked language model task and a well-designed network. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large, achieving state-of-the-art results among unsupervised sentence representation learning methods.

## introduction sentence embeddings aim to capture rich semantic information to be applied in many downstream tasks @xcite @xcite . recently, researchers have started to use contrastive learning to learn better unsupervised sentence embeddings @xcite @xcite @xcite . contrastive learning methods assume that effective sentence embeddings should bring similar sentences closer and dissimilar sentences farther. these methods use various data augmentation methods to randomly generate different * the first two authors contribute equally. ## infocse: information-aggregated contrastive learning in this section, we first introduce the mlm pretraining of the auxiliary network, and then describe how to jointly train the contrastive learning objective and the mlm objective with the auxiliary network. ## ablation studies we perform an extensive series of ablation studies to infocse with bert base scale on the development set of sts-b. ## analysis in this section, we further analyze the performance of infocse on transfer tasks and in-domain information retrieval task. we also explore whether different auxiliary objectives can coexist. ## related work unsupervised sentence representation learning has been widely studied. @xcite propose to learn sentence representation according to the internal structure of each sentence. @xcite predict the surrounding sentences of a given sentence based on the distribution hypothesis. @xcite propose sent2vec, a simple unsupervised model allowing to compose sentence embeddings using word vectors along with n-gram embeddings. recently, contrastive learning has been explored in unsupervised sentence representation learning and has become a promising trend @xcite @xcite @xcite @xcite @xcite . those contrastive learning based methods for sentence embeddings are generally based on the assumption that a good semantic representation should be able to bring similar sentences closer while pushing away dissimilar ones. the most related ones are @xcite and @xcite .we solve the problem in @xcite that the mlm task will reduce the model effect on sts through designing an auxiliary network. moreover, we achieve further improvement on sts via information aggregation. @xcite is similar to us in network structure, and we both design an auxiliary network. but the motivations are quite different. @xcite aims to introduce equivariant contrastive learning to simcse. then the sentence representations will be aware of, but not necessarily invariant to, direct surface-level augmentations. we aim to simultaneously achieve information aggregation while solving the compatibility problem of mlm and contrastive learning. ## conclusion and future work in this paper, we present infocse, an informationaggregated contrastive learning framework for learning unsupervised sentence embeddings. empirical improvements on different datasets show the effectiveness and transferability of infocse. we also conduct extensive ablation studies to demonstrate the different modeling choices in infocse. we believe that improving sentence embeddings by optimizing auxiliary objectives is a very promising direction. in the future, we will further explore its potential on more tasks, such as dense passage retrieval @xcite .