Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity (2024)

Abstract: Semantic Textual Similarity (STS) constitutes a critical research direction in computational linguistics and serves as a key indicator of the encoding capabilities of embedding models. Driven by advances in pre-trained language models and contrastive learning, leading sentence representation methods have reached an average Spearman’s correlation score of approximately 86 across seven STS benchmarks in SentEval. However, further progress has become increasingly marginal, with no existing method attaining an average score higher than 86.5 on these tasks. This paper conducts an in-depth analysis of this phenomenon and concludes that the upper limit for Spearman’s correlation scores under contrastive learning is 87.5. To transcend this ceiling, we propose an innovative approach termed Pcc-tuning, which employs Pearson’s correlation coefficient as a loss function to refine model performance beyond contrastive learning. Experimental results demonstrate that Pcc-tuning can markedly surpass previous state-of-the-art strategies with only a minimal amount of fine-grained annotated samples.

## introduction as a fundamental task within natural language processing (nlp), semantic textual similarity (sts) is not only widely applied across various real-world scenarios including text clustering, information retrieval, and dialogue systems, but also serves as a principal means for evaluating sentence embeddings @xcite . sentence embeddings refer to vector encodings that encapsulate the semantic essence of original texts. owing to their capacity to facilitate offline computation as well as their pivotal role in realizing retrieval-augmented generation @xcite , research in this area has garnered considerable attention from numerous institutions and scholars in recent years. the quality of sentence embeddings is typically assessed via the senteval @xcite toolkit, which measures models based on their average spearman correlation across seven sts benchmarks. with the continuous advancement of pre-trained language models (plms), contrastive learning, and prompt engineering, cuttingedge works in this field have progressively elevated leaderboard scores from an initial 60 @xcite to around 86 @xcite . as a result, the "plm + contrastive learning" framework has become the mainstream paradigm in sentence representation research. however, as illustrated in in this regard, deelm @xcite posits that plms may have reached their performance limits on sts tasks. however, this paper will demonstrate through rigorous mathematical derivation that the core factor causing this performance ceiling is not the inadequacy of plms, but inherent flaws in contrastive learning loss functions. specifically, contrastive learning only distinguishes between two categories: similar and dissimilar, in determining the semantic relationships between text pairs. this binary classification strategy restricts its maximum achievable spearman's correlation score @xcite . ♠: results from @xcite . to 87.5, even under optimal conditions. following this proof, we introduce pcc-tuning, a novel approach that employs a two-stage training process. this method enhances models' semantic discrimination capabilities by leveraging a small amount of fine-grained annotated data post contrastive learning. with the same 7b-scale generative plms, pcc-tuning can substantially surpass previous best results on the seven aforementioned sts tasks and break through the performance ceiling of 87.5. the main contributions of this study are outlined as follows: ## proposed method this section introduces pcc-tuning, an innovative strategy for addressing sts tasks. pcc-tuning employs a two-stage training pipeline and is designed to break the 87.5 performance ceiling in contrastive learning methods. the anisotropy of plms' semantic space (ethayarajh, 2019) is a longstanding challenge in sentence representation research. contrastive learning has proven effective in stabilizing embedding distances among semantically similar texts while promoting a more uniform distribution of overall vector encodings @xcite , thus markedly enhancing the semantic properties of plms. therefore, leveraging contrastive learning to refine the initial state of pre-trained models has emerged as a prevalent approach within the nlp community @xcite @xcite . following this well-established practice, we initially conduct supervised fine-tuning of the plm using the nli dataset constructed by simcse @xcite . this dataset comprises 275,601 tripletform text pairs, providing a robust source of coarsegrained labeled information for the model. our implementation in the first stage closely mirrors that of prompteol @xcite for comparison purposes, where we load the original plm checkpoint and fine-tune the model with the extended infonce loss depicted in equation 7, combined with qlora @xcite . a distinctive feature of our methodology is the adoption of the promptsum template proposed by @xcite : "this sentence : '[x]' can be summarized as", which encapsulates the input sentence [x] and extracts the encoding of the final token as the sentence embedding. later sections will examine pcc-tuning's performance under various prompts. after the contrastive learning phase, the plm will be adjusted to a superior encoding state, capable of producing high-quality embeddings. however, the neural network trained at this stage re-mains insufficient as the final solution for sts tasks. this is due to two primary reasons: (1) the contrastive learning objective does not fully align with the evaluation metrics of sts tasks. while a decrease in infonce loss reflects a better clustering effect of sentence vectors in semantic space, this does not necessarily translate into an improvement in spearman correlation. the latter essentially measures the consistency of the model's scoring with human ratings in terms of monotonicity. the optimal strategy is to incorporate finegrained annotated data in the second stage and guide the model's training process via spearman's correlation coefficient. this ensures maximum con-sistency between the model's behavior during training and testing. however, since spearman correlation is non-differentiable and thus incompatible with backpropagation, we opt for pearson's correlation coefficient to update model parameters, which also serves as the inspiration for the name pcctuning. pearson correlation and our loss function for the second stage are shown in equation 8 , where x represents the cosine similarity between modelderived embeddings, and y denotes the humanannotated scores for the text pairs. @xmath0, y) concretely, for a batch of text pairs {(x i , x ? employing pearson's correlation coefficient as the loss function enables effective utilization of fine-grained sample scores and supports diverse combinations even with relatively small data quantities. for example, our tuning dataset in the second stage is composed of filtered training sets from sts-b @xcite and sick-r @xcite , which together contain 5,398 text pairs. this number merely constitutes 1.96% of the size of the nli dataset adopted in the first stage, yet the potential combination varieties reach up to c n 5398 (where n represents the batch size). as a result, even after multiple epochs of training, the similarity rankings of samples in each batch are unlikely to repeat, thereby continuously providing the model with meaningful gradient information. the dataset is filtered because we discovered that some sentence pairs in the sts-b and sick-r training sets overlap with the test sets of the seven sts benchmarks in senteval. additionally, there is even overlap between the train and test sets within sick-r itself. to prevent information leakage, we implemented a stringent filtering mechanism to ensure that the model does not encounter any test set text pairs during parameter updates. more details about this filtering process can be found in appendix b. figure 2 presents an overview of pcc-tuning's training pipeline. in the first stage, the model is fine-tuned using contrastive learning on the nli corpus. in the second stage, we introduce a small amount of fine-grained annotated data and load the checkpoint from the first phase to further update the model parameters via pearson's correlation coefficient. this two-stage fine-tuning strategy effectively prevents overfitting. although the filtered sts-b and sick-r training sets provide only 5,398 fine-grained labeled instances, the 275,601 text pairs used in the first stage establish a solid initial state for the model, thereby maximizing its generalization capacity. moreover, the varying scoring scales of sts-b and sick-r also introduce a degree of noise into the pcc-tuning training process , which contributes to the model's robustness. we provide detailed results of our method's zero-shot performance on several downstream tasks in appendix c to further demonstrate its transferability. ## experiments this section presents the experimental results of pcc-tuning. initially, in subsection 4.1, we elaborate on our experimental setup, including evaluation methods, training data size, and the selection of baselines. subsequently, in subsection 4.2, we compare the performance of pcc-tuning with current sota text representation strategies across internationally recognized sts benchmarks. following this, in subsection 4.3, we conduct targeted experiments to demonstrate that pcc-tuning surpasses continuous contrastive learning. finally, in subsection 4.4, we examine the efficacy of pcc-tuning under different prompts. ## related work contrastive learning is currently the principal strategy within the nlp community for addressing sts tasks, and our proposed method, pcc-tuning, is specifically designed to overcome the inherent limitations of these approaches. prior to the rise of contrastive learning-based text representation schemes, sentence-bert had already introduced the idea of enhancing the semantic encoding capabilities of plms using the sts-b training set @xcite . however, subsequent contrastive learning methods, such as simcse @xcite , promptbert @xcite , and cot-bert @xcite have demonstrated superior performance across the seven sts benchmarks collected in sen-teval, thereby making them the focal point of re- to the best of our knowledge, this study is the first to propose and substantiate the theoretical performance upper bound of contrastive learning methods. additionally, pcc-tuning is the inaugural method capable of achieving spearman's correlation scores above 87 on standard sts tasks, marking a significant advancement in the field. ## conclusion in this paper, we first analyze the structure of contrastive learning loss functions, highlighting that their coarse-grained categorization of semantic relationships among texts renders contrastive learning akin to a binary classifier. building on this insight, we rigorously derive the optimal spearman correlation achievable by a binary classifier in sts tasks, establishing that the upper bound for the spearman correlation of contrastive learning methods is 87.5. to achieve further breakthroughs, we introduce pcc-tuning, a novel strategy that effectively harnesses fine-grained annotated information. pcctuning leverages a two-stage training pipeline and utilizes pearson's correlation coefficient as the loss function to fully exploit the ordinal relationships between text pairs. extensive experimental results demonstrate that pcc-tuning significantly enhances the quality of generated embeddings, with consistent performance gains observed across various plms, prompts, and batch sizes.