Pairwise Supervised Contrastive Learning of Sentence Representations (2021)

Abstract: Many recent successes in sentence representation learning have been achieved by simply fine-tuning on the Natural Language Inference (NLI) datasets with triplet loss or siamese loss. Nevertheless, they share a common weakness: sentences in a contradiction pair are not necessarily from different semantic categories. Therefore, optimizing the semantic entailment and contradiction reasoning objective alone is inadequate to capture the high-level semantic structure. The drawback is compounded by the fact that the vanilla siamese or triplet losses only learn from individual sentence pairs or triplets, which often suffer from bad local optima. In this paper, we propose PairSupCon, an instance discrimination based approach aiming to bridge semantic entailment and contradiction understanding with high-level categorical concept encoding. We evaluate PairSupCon on various downstream tasks that involve understanding sentence semantics at different granularities. We outperform the previous state-of-the-art method with 10%–13% averaged improvement on eight clustering tasks, and 5%–6% averaged improvement on seven semantic textual similarity (STS) tasks.

## introduction learning high-quality sentence embeddings is a fundamental task in natural language processing. the goal is to map semantically similar sentences close together and dissimilar sentences farther apart in the representation space. many recent successes have been achieved by training on the nli datasets @xcite @xcite , where the task is often to classify each sentence pair into one of three categories: entailment, contradiction, or neutral. despite promising results, prior work @xcite @xcite share a common weakness: the sentences forming a contradiction pair may not necessarily belong to different semantic categories. consequently, optimizing the model for semantic entailment and contradiction understanding alone is inadequate to encode the high-level categorical concepts into the representations. moreover, the vanilla siamese (triplet) loss only learns from the individual sentence pairs (triplets), which often requires substantial training examples to achieve competitive performance @xcite . as shown in section 4.1, the siamese loss can sometimes drive a model to bad local optima where the performance of high-level semantic concept encoding is degraded when compared with its baseline counterpart. in this paper, we take inspiration from selfsupervised contrastive learning @xcite @xcite and propose jointly optimizing the pairwise semantic reasoning objective with an instance discrimination loss. we name our approach pairwise supervised contrastive learning (pairsupcon). as noticed by the recent work @xcite , instance discrimination learning can implicitly group similar instances together in the representation space without any explicit learning force directs to do so. pairsupcon leverages this implicit grouping effect to bring together representations from the same semantic category while, simultaneously enhancing the semantic entailment and contradiction reasoning capability of the model. although the prior work mainly focuses on pairwise semantic similarity related evaluations, we argue in this paper that the capability of encoding the high-level categorical semantic concept into the representations is an equally important aspect for evaluations. as shown in section 4, the previous state-of-the-art model that performs best on the semantic textual similarity (sts) tasks results in degenerated embeddings of the categorical semantic structure. on the other hand, better capturing the high-level semantic concepts can in turn promote better performance of the low-level semantic entailment and contradiction reasoning. this assumption is consistent with how human categorize objects in a top-down reasoning manner. we further validate our assumption in section 4, where pairsupcon achieves an averaged improvement of 10% -13% over the prior work when evaluated on eight short text clustering tasks, and yields 5% -6% averaged improvement on seven sts tasks. ## related work sentence representation learning with nli the suitability of leveraging nli to promote better sentence representation learning is first observed by infersent @xcite , where a siamese bilstm network is optimized in a supervised manner with the semantic entailment and contraction classification objective. universal sentence encoder @xcite later augments an unsupervised learning objective with the supervised learning on nli, and shows better transfer performance on various downstream tasks. more recently, sbert @xcite finetunes a siamese bert @xcite model on nli and sets new state-of-the-art results. however, sbert as well as the above work adopt the vanilla siamese or triplet loss, which often suffer from slow convergence and bad local optima @xcite . ## model following sbert @xcite , we adopt the snli @xcite and mnli @xcite as our training data, and refer the combined data as nli for convenience. the nli data consists of labeled sentence pairs and each can be presented in the form: (premise, hypothesis, label). the premise sentences are selected from existing text sources and each premise sentence is paired with variant hypothesis sentences composed by human annotators. each label indicates the hypothesis type and categorizes semantic relation of the associated premise and hypothesis sentence pair into one of three categories: entailment, contradiction, and neural, correspondingly. prior work solely optimizes either siamese loss or triplet loss on nli. we instead aiming to leverage the implicit grouping effect of instance discrimination learning to better capture the high-level categorical semantic structure of data while, simultaneously promoting better convergence of the lowlevel semantic textual entailment and contradiction reasoning objective. ## experiments baselines in this section, we mainly investigate the effective strategies of leveraging the labeled nli data to enhance the sentence representations of the pre-trained language models (plms). we compare pairsupcon against the vanilla bert @xcite models and the previous state-of-the-art approach, sbert @xcite . we noticed a concurrent work simcse @xcite ## discussion and conclusion in this paper, we present a simple framework for universal sentence representation learning. we leverage the implicit grouping effect of instance discrimination learning to better encoding the highlevel semantic structure of data into representations while, simultaneously promoting better convergence of the lower-level semantic entailment and contradiction reasoning objective. we substantially advance the previous state-of-the-art results when evaluated on various downstream tasks that involve understanding semantic concepts at different granularities. we carefully study the key components of our model and pinpoint the performance gain contributed by each of them. we observe encouraging performance improvement by using the proposed hard negative sampling strategy. on the other hand, hard negative sampling without any supervision is an crucial, yet significantly challenging problem that should motivate further explorations. possible directions include making proper structure assumption or leveraging domain-specific knowledge. the substantial performance gain attained by our model also suggests developing explicit grouping objectives could be another direction worth investigation.