Ranking-Enhanced Unsupervised Sentence Representation Learning (2023)

Abstract: Unsupervised sentence representation learning has progressed through contrastive learning and data augmentation methods such as dropout masking. Despite this progress, sentence encoders are still limited to using only an input sentence when predicting its semantic vector. In this work, we show that the semantic meaning of a sentence is also determined by nearest-neighbor sentences that are similar to the input sentence. Based on this finding, we propose a novel unsupervised sentence encoder, RankEncoder. RankEncoder predicts the semantic vector of an input sentence by leveraging its relationship with other sentences in an external corpus, as well as the input sentence itself. We evaluate RankEncoder on semantic textual benchmark datasets. From the experimental results, we verify that 1) RankEncoder achieves 80.07% Spearman’s correlation, a 1.1% absolute improvement compared to the previous state-of-the-art performance, 2) RankEncoder is universally applicable to existing unsupervised sentence embedding methods, and 3) RankEncoder is specifically effective for predicting the similarity scores of similar sentence pairs.

## introduction sentence representation learning aims to encode sentences into a semantic vector space. this task has been a fundamental task in natural language processing (nlp), as universal sentence vectors are widely applicable to many nlp tasks @xcite @xcite @xcite . recently, unsupervised sentence embedding methods have arisen as they have shown a potential to overcome limited labeled data with simple data augmentation methods @xcite figure 1 : vector representations of sentences and their neighbor sentences. the neighbor sentences reveal that (a, c) share more semantic meanings than (a, b). this captures more accurate semantic similarity scores than their vectors. 2021; @xcite @xcite @xcite . these approaches minimize the distance between the vector representations of similar sentences, called positive pairs, while maximizing the distance between those of dissimilar sentences, called negative pairs. many studies have focused on developing better positive and negative pair sampling methods. data augmentation methods such as dropout masking @xcite , token shuffling @xcite , and sentence negation @xcite have been proposed and achieved comparable semantic textual similarity performance to sentence encoders trained on human-annotated datasets. the semantic meaning of a sentence is not only determined by the words within the sentence itself but also by other sentences with similar meanings. however, previous unsupervised sentence embed-ding methods use only the input sentence when predicting its semantic vector. figure 1 shows example sentences and their semantic vector space. in this figure, the human-annotated similarity scores indicate that sentence pair (a, c) is more similar than (a, b). however, the similarity scores computed by their sentence vectors indicate the opposite result; the vector representations of a and b are closer than a and c as they have more overlapping words than a and c. this problem can be alleviated by leveraging the distance between the input sentence and other sentences in a large corpus. the vectors of their neighbor sentences approximate the overall semantic distribution, and the semantic distribution reveals that sentences a and c are likely to share more semantic meanings than sentences a and b. this facilitates the accurate prediction of semantic similarity between sentences. in this paper, we propose rankencoder, a novel unsupervised sentence encoder that leverages a large number of sentences in an external corpus. for a given corpus with n sentences and an input sentence, rankencoder computes a rank vector, an n-dimensional vector in which i'th element represents the distance between the input sentence and i'th sentence in the corpus; rankencoder uses an existing unsupervised sentence encoder, e, to compute the distances. then, two sentences that share the same neighbor sentences (e.g., sentence a and c in fig 1 ) have similar rank vector representations. we verify that using the similarity between rank vectors captures better semantic similarity than their vector representation computed by the base encoder, e, (fig 4 ) without further training. we further leverage the similarity scores predicted by the rank vectors to train another sentence encoder and achieve a better sentence encoder ( from experiments on seven sts benchmark datasets, we verify that 1) rank vectors are effective for capturing the semantic similarity of similar sentences, 2) rankencoder is applicable to any unsupervised sentence encoders, resulting in performance improvement, and 3) this improvement is also valid for the previous state-of-the-art sentence encoder and leads to a new state-of-the-art semantic textual similarity performance. first, we measure the performance of rankencoder and the baselines on three sentence pair groups divided by their similarity scores. the experimental results show that rankencoder is effective on similar sentence pairs. second, we apply rankencoder to the three unsupervised sentence encoders, simcse @xcite , promptbert @xcite , and sncse @xcite , then verify that our approach brings performance improvement to each encoder. third, we apply rankencoder to the stateof-the-art unsupervised sentence encoder @xcite and achieve a 1.1% improvement; the previous state-of-the-art is 78.97 spearman's correlation, and we achieve 80.07 spearman's correlation. the contributions of this paper are three folds. first, we demonstrate that the semantic meaning of a sentence is also determined by its nearestneighbor sentences as well as the words within the sentence itself. second, we propose rankencoder, which leverages a large number of sentences to capture the semantic meanings of sentences. third, we achieve state-of-the-art sts performance and reduce the gap between supervised and unsupervised sentence encoders; the performances of our method and the state-of-the-art supervised sentence encoder @xcite are 80.07 and 81.97, respectively. ## related works unsupervised sentence representation learning has progressed through contrastive learning with positive and negative sentence pair sampling methods @xcite @xcite . simcse @xcite and consert @xcite apply data augmentation methods such as dropout masking, token shuffling, and adversarial attacks to an input sentence and sample a positive pair. however, these data augmentation methods often change the meaning of the input sentence and generate dissimilar positive pairs. a masked language modeling-based word replacement method has been proposed to alleviate this problem @xcite . they train a sentence encoder to predict the replaced words and make the encoder aware of surface-level augmentations. some studies adopt momentum contrastive learning to generate positive samples inspired by unsupervised visual representation learning @xcite . prompting @xcite is another direction that is capable of generating positive pairs. recently, a negative sampling method for data augmentation has been proposed @xcite . this approach takes the negation of an input sentence and uses this soft-negative sample in a contrastive learning framework. compared to previous approaches focused on generating better positive and negative pairs, our work uses nearestneighbor sentences to predict better semantic vectors of sentences, a novel approach that previous approaches have yet to cover. related but different from our work, trans-encoder proposes the selfdistillation method that gets supervision signals from itself @xcite . trans-encoder solves a slightly different problem from ours. they aim to solve an unsupervised sentence pair modeling problem, not unsupervised sentence embedding; although this work does not require any humanannotated similarity scores of sentence pairs for training, they need the sentence pairs of the sts datasets, which are not allowed to be used for training in unsupervised sentence representation learning. ## method leveraging the k-nearest-neighbor sentences helps a sentence encoder to approximate a more accurate semantic meaning of a sentence. for instance, when two input sentences have more common neighbors than other sentences, it is likely that they are semantically similar; we have provided an example in figure 1 . we extend this idea to leverage the entire sentences in the corpus, not just the neighbor sentences. our unsupervised sentence encoder, rankencoder, computes a rank vector for a given sentence. the rank vector is a list of ranks of all sentences in the corpus computed by their similarity scores to the input; for a given corpus with n number of sentences, a rank vector is an n-dimensional vector, in which i'th element represents the rank of i'th sentence in the corpus. thus, when two input sentences have common neighbor sentences, their rank vectors are similar. we found that rank vectors capture more accurate semantic similarity than previous unsupervised sentence encoders. since rank vectors predict better semantic similarity scores between sentences, we use these scores for training another sentence encoder to further increase its sts performance. we provide the overall illustration of rankencoder in figure 2 . ## results and discussions in this section, we demonstrate that 1) rankencoder is effective for capturing the semantic similarity scores of similar sentences, 2) rankencoder is universally applicable to existing unsupervised sentence encoders, and 3) rankencoder achieves state-of-the-art semantic textual similarity (sts) performance. we describe the detailed experimental results in the following sections. ## conclusion in this study, we showed that the semantics of a sentence is also determined by its similar sentence, not just the words within the sentence itself. we proposed rankencoder to overcome the limitation of the previous sentence representation learning approaches, which are limited to using only the input sentence. rankencoder leverages the distance between the input sentence and the sentences in a corpus to predict its semantic vector. rankencoder is universally applicable to any unsupervised sentence encoder, resulting in performance improvement, and we demonstrated this with three unsupervised sentence encoders. we achieved state-of-the-art semantic textual similarity performance by applying our approach to the previous best sentence encoder. we also showed that our approach is specifically effective for capturing the semantic similarities of similar sentences.