R ank CSE : Unsupervised Sentence Representations Learning via Learning to Rank (2023)

Abstract: Unsupervised sentence representation learning is one of the fundamental problems in natural language processing with various downstream applications. Recently, contrastive learning has been widely adopted which derives high-quality sentence representations by pulling similar semantics closer and pushing dissimilar ones away. However, these methods fail to capture the fine-grained ranking information among the sentences, where each sentence is only treated as either positive or negative. In many real-world scenarios, one needs to distinguish and rank the sentences based on their similarities to a query sentence, e.g., very relevant, moderate relevant, less relevant, irrelevant, etc. In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. In particular, we learn semantically discriminative sentence representations by simultaneously ensuring ranking consistency between two representations with different dropout masks, and distilling listwise ranking knowledge from the teacher. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks. Experimental results demonstrate the superior performance of our approach over several state-of-the-art baselines.

## introduction to obtain semantically discriminative sentence representations, we propose a novel approach, rankcse, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. specifically, our model ensures ranking consistency between two representations with different dropout masks and minimizes the jensen-shannon (js) divergence as the learning objective. in the meanwhile, our model also distills listwise ranking knowledge from the teacher model to the learned sentence representations. in our work, we explore two listwise ranking methods, listnet @xcite and listmle @xcite , and utilize the pre-trained simcse @xcite models with coarse-grained semantic ranking information as the teachers to provide pseudo ranking labels. our rankcse is able to generalize fine-grained ranking information from the weak ranking knowledge learned by simcse. we conduct an extensive set of experiments on semantic textual similarity (sts) and transfer (tr) tasks. experimental results show that rankcse outperforms the existing state-of-the-art baselines. recently, contrastive learning has been adopted to learn sentence representations by designing different augmentation methods @xcite @xcite @xcite . a typical example simcse uses dropout as a data augmentation strategy and is also the foundation of many following works. arccse @xcite enhances the pairwise discriminative power and models the entailment relation among triplet sentences. dclr @xcite alleviates the influence of improper negatives. diffcse @xcite introduces equivariant contrastive learning to sim-cse. pcl @xcite proposes contrastive representation learning with diverse augmentation strategies for an inherent anti-bias ability. infocse @xcite ) learns sentence representations with the ability to reconstruct the original sentence fragments. generative learning techniques @xcite have also been proposed to enhance the linguistic interpretability of sentence representations. although achieving promising results, these methods fail to capture the fine-grained ranking knowledge among the sentences. learning to rank given a query example, learning to rank aims to rank a list of examples according to their similarities with the query. learning to rank methods can be divided into three categories: pointwise @xcite , pairwise @xcite and listwise @xcite @xcite . pointwise methods optimize the similarity between the query and each example, while pairwise approaches learn to cor-rectly model the preference between two examples. listwise methods directly evaluate the ranking of a list of examples based on the ground truth. in our framework, we leverage listwise ranking objectives for learning effective sentence representations, which have shown better performance compared to pointwise and pairwise methods. ## preliminary we provide some conceptual explanations and definitions in learning to rank. ## conclusion in this work, we propose rankcse, an unsupervised approach to learn more semantically discriminative sentence representations. the core idea of rankcse is incorporating ranking consistency and ranking distillation with contrastive learning into a unified framework. when simultaneously ensuring ranking consistency and distilling listwise ranking knowledge from the teacher, rankcse can learn how to make fine-grained distinctions in semantics, leading to more semantically discriminative sentence representations. experimental results on sts and tr tasks demonstrate that rankcse outperforms previous state-of-the-art methods. we also conduct thorough ablation study and analysis to demonstrate the effectiveness of each component and justify the inner workings of our approach. we leave what is the upper bound of improvements of the teacher for future work.