A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space (2022)

Abstract: Learning high-quality sentence representations is a fundamental problem of natural language processing which could benefit a wide range of downstream tasks. Though the BERT-like pre-trained language models have achieved great success, using their sentence representations directly often results in poor performance on the semantic textual similarity task. Recently, several contrastive learning methods have been proposed for learning sentence representations and have shown promising results. However, most of them focus on the constitution of positive and negative representation pairs and pay little attention to the training objective like NT-Xent, which is not sufficient enough to acquire the discriminating power and is unable to model the partial order of semantics between sentences. So in this paper, we propose a new method ArcCSE, with training objectives designed to enhance the pairwise discriminative power and model the entailment relation of triplet sentences. We conduct extensive experiments which demonstrate that our approach outperforms the previous state-of-the-art on diverse sentence related tasks, including STS and SentEval.

## introduction learning sentence representations, which encodes sentences into fixed-sized dense vectors such that semantically similar ones stay close, is a fundamental problem of natural language processing. it could benefit a wide range of downstream applications such as information retrieval, semantic similarity comparison, question answering, and so on. recently, with the great success of pre-trained transformer-based language models @xcite @xcite @xcite like bert, they have been widely adopted for generating sentence representations. a straightforward way is by leveraging the [cls] embedding @xcite -1 0 1 to make pre-trained language models more suitable for generating sentence embeddings, supervised methods like sbert @xcite are proposed, which improve the performance by fine-tuning on a labeled dataset. as labeled data is not available or expensive to annotate in many tasks or domains, it is of great value for developing unsupervised/self-supervised approaches for learning sentence representations. so recent works like bert-flow @xcite and bert-whitening @xcite propose postprocessing methods to improve the bert-based sentence representation. they address that the nonsmooth anisotropic semantic space of bert is a bottleneck and alleviate the problem through normalizing flows and whitening operation. to further improve the quality of sentence representations, several works @xcite @xcite @xcite adopt self-supervised contrastive learning approach, which learns sentence representations by minimizing the distance of positive sentence representation pairs and maximizing the distance of negative pairs. in these works, positive pairs are often constituted through data augmentation or encoders with different structure or parameters, while negative pairs are derived from different sentences within the same batch. then contrastive learning objective like normalized temperature-scaled crossentropy loss (nt-xent) @xcite is used for optimizing. a typical example unsup-simcse @xcite achieves state-of-the-art performance with a simple and effective idea of using standard dropout for data augmentation. though existing contrastive methods for learning sentence representation have shown promising results, most of them focus on the positive and negative pairs constitution, and the optimization objective itself is not fully exploited. the contrastive learning objective nt-xent loss used in recent works @xcite @xcite is a variation of cross-entropy loss with softmax function. recent studies @xcite have shown that the traditional softmax-based loss is insufficient to acquire the discriminating power, as shown in figure 1(b) in which simcse-bert base adopts the nt-xent loss and could not separate s b and s c completely. in addition, the current optimization objectives only models sentence relations in a pairwise perspective, which tries to pull sentences with similar semantics closer and push dissimilar ones away from each other. however, there are different degrees of semantic similarity among related sentences. for example in figure 1 (d), s b is more similar to s a than s c is. the current optimization objectives lack the ability to model the partial order of semantics between sentences. to alleviate these problems, in this paper, we propose a new approach arccse for sentence representation learning. for pairwise sentence relation modeling, we propose additive angular margin contrastive loss (arccon loss), which enhances the pairwise discriminative power by maximizing the decision margin in the angular space. besides, in order to model the partial order of semantics between sentences, we propose a new self-supervised task that captures the entailment relation among triplet sentences. the task is implemented through automatically constituted triplet sentences with entailment relation among them. a visualization example of the generated representations through arccse is shown in figure 1(c ). we evaluate our method on standard semantic textual similarity (sts) tasks and senteval transfer tasks, and it outperforms the previous state-of-the-art approaches. ## method in this section, we present arccse, an angular based contrastive sentence representation learning framework, which could generate superior sentence embeddings from unlabeled data. given a pretrained language model m and an unlabeled text dataset d, the task is fine-tuning m on d so that the sentence representations generated through m could be more semantic discriminative. our framework consists of two components that model pairwise and triple-wise sentence relations simultaneously, as shown in figure 2 . we start with angular margin based contrastive learning in section 3.1, which models pairwise relations between sentences by pulling semantic similar ones closer while pushing dissimilar ones away. then we introduce the method which models the partial order of semantics between automatically constituted triplet sentences in section 3.2. ## conclusion in this work, we propose arccse, a self-supervised contrastive learning framework for learning sentence representation. we propose a new optimizing objective arccon loss to model pairwise sentence relations with enhanced discriminating power, and a new self-supervised task to model the partial order of semantics between sentences. experimental results on semantic textual similarity tasks (sts) and senteval tasks demonstrate that both techniques bring substantial improvements and our method outperforms previous state-of-theart method for sentence representation learning.