Contrastive Learning of Sentence Representations (2021)

Abstract: Learning sentence representations which capture rich semantic meanings has been crucial for many NLP tasks. Pre-trained language models such as BERT have achieved great success in NLP, but sentence embeddings extracted directly from these models do not perform well without fine-tuning. We propose Contrastive Learning of Sentence Representations (CLSR), a novel approach which applies contrastive learning to learn universal sentence representations on top of pre-trained language models. CLSR utilizes semantic similarity of two sentences to construct positive instance for contrastive learning. Semantic information that has been captured by the pre-trained models is kept by getting sentence embeddings from these models with proper pooling strategy. An encoder followed by a linear projection takes these embeddings as inputs and is trained under a contrastive objective. To evaluate the performance of CLSR, we run experiments on a range of pre-trained language models and their variants on a series of Semantic Contextual Similarity tasks. Results show that CLSR gains significant performance improvements over existing SOTA language models.

## introduction learning sentence representations that can encode semantic information is crucial for many natural language processing (nlp) tasks such as question answering, summarization, machine translation. many attempts have been made to learn general purpose sentence embeddings @xcite @xcite @xcite @xcite @xcite . since transformerbased pre-trained language models such as bert are introduced and achieve the state-of-the-art results in many nlp tasks, several methods have been proposed to generate sentence embeddings with some pooling strategy such as mean, max from word level embeddings and fine tune these models on downstream tasks @xcite or train to calibrate these models for isotropic embeddings @xcite . inspired by the recent development of contrastive learning in learning visual representations @xcite , we design a contrastive learning based architecture to learn high-quality semantic sentence representations and show this approach can significantly improve the sentence representations. we integrate both pre-trained language models and contrastive learning and call our architecture contrastive learning of sentence representations (clsr). different from previous methods of using data augmentation to construct positive pairs in contrastive learning @xcite , we use semantic similarity or entailment relation of two sentences to build positive pairs. by sending two similar sentences into a pre-trained language model and then generate vector representations by pooling, clsr is able to keep the rich information captured by these pre-trained models. a contrastive objective is used to train an encoder followed by a linear projection to further learn the embeddings in an unsupervised way. clsr is model-agnostic. the initial sentence embeddings it takes as inputs can come from any pre-trained model. ## related work learning sentence embeddings has attracted a lot of interest. paragraph vector @xcite and skip-thought @xcite learns generic, distributed sentence representations in an unsupervised fashion, one by proposing two log-bilinear models and the other one by training an encoder-decoder model to reconstruct the surrounding sentences of an encoded passage. @xcite proposed sequential denoising autoencoders and fastsent to learn sentence representations from unlabelled data. infersent @xcite performs the learning in a supervised way by training a siamese bilstm network on stanford natural language inference (snli) dataset @xcite . universal sentence encoder @xcite , with two variants, combines both by training with transfer learning on unsupervised data and being augmented on supervised data from the snli corpus @xcite . recently there have been several attempts to improve sentence embeddings from pooled outputs of pre-trained language model such as bert @xcite . sentence-bert @xcite trains a siamese network to finetune bert and its variants. bert-flow @xcite learns more smooth and isotropic embeddings by applying normalizing flow @xcite to convert bert sentence embedding distribution into a gaussian distribution. sbert-wk @xcite improves sentence-bert by incorporating the pattern of layer-wised word representations in subspace. @xcite applied whitening technique to enhance the isotropy of sentence representations. since contrastive learning has achieved great success in unsupervised representation learning in computer vision, it also has gained much interest in nlp including sentence representation learning. the following are some recent or concurrent work, many of which have explored different data augmentation strategies. is-bert @xcite learns through maximizing the mutual information between the global sentence representation and its local token representation. cert (fang and xie, 2020) augments sentences using back-translation @xcite . declutr @xcite applies a contrastive objective on textual segments sampled from nearby in the same document. clear @xcite uses multi sentence-level augmentation to construct positive pairs. @xcite proposes contrast tension which counters the task biases in pre-trained language models by contrasting the noise between the output from two independent models. consert @xcite explores several ways to augment data such as adversarial attack, token shuffling etc. simcse @xcite constructs positive instances by taking different outputs of the same sentence from the same pre-trained language model using dropout. besides treating the task as unsupervised learning, although some of the above work such as @xcite , @xcite also explore it as supervised learning, our method is mainly to show, by simply using sentence pairs with high similarity or entailment relation in existing labeled corpus to construct positive instances, contrastive learning can still further significantly improve the quality of sentence embeddings on top of any pre-trained language model. clsr adopts contrastive learning framework simclr @xcite to learn sentence semantic representations as shown in figure 1 . clsr consists of: after randomly sampling m sentence pairs into a mini batch, this batch contains 2m sentences. each sentence pair is treated as a positive pair, and the rest of sentences in the batch are treated as inbatch negatives @xcite . we hypothesize that the probability of having one or more sentences in the negative samples that are highly semantically similar with that in the positive pair is very low and thus is ignored. ## conclusion and future work this paper presents a novel approach of applying contrastive learning on pre-trained language models to learn generic sentence representations. the evaluation on a series of sts tasks shows that our approach outperforms the pre-trained sota language models significantly. how to construct multiple positive instances and further integrate the idea of contrastive learning will be explored in future.