C on SERT : A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021)

Abstract: Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised SEntence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.

## introduction sentence representation learning plays a vital role in natural language processing tasks @xcite @xcite . good sentence representations benefit a wide range of downstream tasks, especially for computationally expensive ones, including largescale semantic similarity comparison and information retrieval. recently, bert-based pre-trained language models have achieved high performance on many downstream tasks with additional supervision. however, the native sentence representations derived from bert foot_0 are proved to be of low-quality @xcite . as shown in figure 1a , when directly adopt bertbased sentence representations to semantic textual similarity (sts) tasks, almost all pairs of sentences achieved a similarity score between 0.6 to 1.0 , even if some pairs are regarded as completely unrelated by the human annotators. in other words, the bert-derived native sentence representations are somehow collapsed @xcite , which means almost all sentences are mapped into a small area and therefore produce high similarity. such phenomenon is also observed in several previous works @xcite @xcite . they find the word representation space of bert is anisotropic, the high-frequency words are clustered and close to the origin, while low-frequency words disperse sparsely. when averaging token embeddings, those high-frequency words dominate the sentence representations, inducing biases against their real semantics 2 . as a result, it is inappropriate to directly apply bert's native sentence representations for semantic matching or text retrieval. traditional methods usually fine-tune bert with additional supervision. however, human annotation is costly and often unavailable in real-world scenarios. to alleviate the collapse issue of bert as well as reduce the requirement for labeled data, we propose a novel sentence-level training objective based on contrastive learning @xcite . by encouraging two augmented views from the same sentence to be closer while keeping views from other sentences away, we reshape the bert-derived sentence representation space and successfully solve the collapse issue (shown in figure 1b ). moreover, we propose multiple data augmentation strategies for contrastive learning, including adversarial attack @xcite , token shuffling, cutoff @xcite and dropout @xcite , that effectively transfer the sentence representations to downstream tasks. we name our approach consert, a contrastive framework for sentence representation transfer. consert has several advantages over previous approaches. firstly, it introduces no extra structure or specialized implementation during inference. the parameter size of consert keeps the same as bert, making it easy to use. secondly, compared with pre-training approaches, consert is more efficient. with only 1,000 unlabeled texts drawn from the target distribution (which is easy to collect in real-world applications), we achieve 35% relative performance gain over bert, and the training stage takes only a few minutes (1-2k steps) on a single v100 gpu. finally, it includes several effective and convenient data augmentation methods with minimal semantic impact. their effects are validated and analyzed in the ablation studies. our contributions can be summarized as follows: self-supervised objectives for pre-training bert @xcite proposes a bidirectional transformer encoder for language model pre-training. it includes a sentence-level training objective, namely next sentence prediction (nsp), which predicts whether two sentences are adjacent or not. however, nsp is proved to be weak and has little contribution to the final performance @xcite . after that, various self-supervised objectives are proposed for pre-training bert-like sentence encoders. cross-thought @xcite and cmlm @xcite are two similar objectives that recover masked tokens in one sentence conditioned on the representations of its contextual sentences. slm @xcite proposes an objective that reconstructs the correct sentence ordering given the shuffled sentences as the input. however, all these objectives need document-level corpus and are thus not applicable to downstream tasks with only short texts. unsupervised approaches bert-flow @xcite proposes a flow-based approach that maps bert embeddings to a standard gaussian latent space, where embeddings are more suitable for comparison. however, this approach introduces extra model structures and need specialized implementation, which may limit its application. ## approach in this section, we present consert for sentence representation transfer. given a bert-like pretrained language model m and an unsupervised dataset d drawn from the target distribution, we aim at fine-tuning m on d to make the sentence representation more task-relevant and applicable to downstream tasks. we first present the general framework of our approach, then we introduce several data augmentation strategies for contrastive learning. finally, we talk about three ways to further incorporate supervision signals. ## experiments to verify the effectiveness of our proposed approach, we conduct experiments on semantic textual similarity (sts) tasks under the unsupervised and supervised settings. ## conclusion in this paper, we propose consert, a selfsupervised contrastive learning framework for transferring sentence representations to downstream tasks. the framework does not need extra structure and is easy to implement for any encoder. we demonstrate the effectiveness of our framework on various sts datasets, both our unsupervised and supervised methods achieve new state-of-the-art performance. furthermore, few-shot experiments suggest that our framework is robust in the data scarcity scenarios. we also compare multiple combinations of data augmentation strategies and provide fine-grained analysis for interpreting how our approach works. we hope our work will provide a new perspective for future researches on sentence representation transfer.