Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated R e LU and Smooth K2 Loss (2024)

Abstract: Since the introduction of BERT and RoBERTa, research on Semantic Textual Similarity (STS) has made groundbreaking progress. Particularly, the adoption of contrastive learning has substantially elevated state-of-the-art performance across various STS benchmarks. However, contrastive learning categorizes text pairs as either semantically similar or dissimilar, failing to leverage fine-grained annotated information and necessitating large batch sizes to prevent model collapse. These constraints pose challenges for researchers engaged in STS tasks that involve nuanced similarity levels or those with limited computational resources, compelling them to explore alternatives like Sentence-BERT. Despite its efficiency, Sentence-BERT tackles STS tasks from a classification perspective, overlooking the progressive nature of semantic relationships, which results in suboptimal performance. To bridge this gap, this paper presents an innovative regression framework and proposes two simple yet effective loss functions: Translated ReLU and Smooth K2 Loss. Experimental results demonstrate that our method achieves convincing performance across seven established STS benchmarks and offers the potential for further optimization of contrastive learning pre-trained models.

## introduction semantic textual similarity (sts) constitutes a fundamental task in natural language processing, wielding significant influence across a multitude of applications, including text clustering, information retrieval, and recommendation systems. despite the remarkable precision obtained by interactive architectures within these tasks, their inability to support offline computation limits their viability in large-scale text analysis scenarios. in response, the seminal work of sentence-bert @xcite introduces a dual-tower architecture to encode the sentences within a pair separately, thereby facilitating the derivation of independent embeddings. this approach showcases superior efficacy and has rapidly gained widespread acceptance, now serving as a cornerstone for various downstream tasks. consequently, further improvements to sentence-bert hold significant research interest and practical value. nevertheless, the advent of contrastive learning methods, exemplified by simcse @xcite , has led to more pronounced enhancements on renowned english sts benchmarks, such as sts12-16 @xcite @xcite @xcite , sts-b @xcite , and sick-r @xcite . this has shifted the research focus in recent years towards integrating contrastive learning techniques with pre-trained language models (plms) like bert @xcite and roberta @xcite ). an intuitive comparison is that, when employing the nli dataset @xcite as a training corpus, simcse-roberta base attains an average spearman's correlation score of 82.52 across these sts tasks, hugely surpassing the 74.21 achieved by sentence-roberta base . such discernible performance disparity has inadvertently overshadowed the advantages of sentence-bert, especially in terms of data utilization efficiency and computational resource demands. contrastive learning, by its self-supervised nature, predominantly recognizes text pairs as either similar or dissimilar. this binary categorization restricts contrastive learning methods to training on triplet-form data composed of an anchor sentence, a positive instance, and a hard negative instance in supervised settings @xcite . many practical scenarios, however, tend to provide more finely grained labeled data (e.g., highly relevant, moderately relevant, relevant, and irrelevant) @xcite , where contrastive learning approaches can usually only exploit text pairs whose similarity indicators are at the endpoints. furthermore, since contrastive learning enhances model discriminability by treating other samples within the same batch as negative instances, it requires large batch sizes, thereby consuming substantial computational resources. for example, simcse's supervised learning settings include a batch size of 512 and 3 epochs. to accommodate this configuration on consumer-grade gpus, sim-cse limits the maximum input length to 32 tokens @xcite . in contrast, sentence-bert and our proposed methodology necessitate a mere batch size of 16 and 1 epoch to reach convergence. additionally, our default maximum input length is 256, significantly longer than simcse's. the aforementioned drawbacks highlight the difficulty in completely replacing sentence-bert with contrastive learning methods. hence, some cutting-edge works @xcite continue to rely on sentence-bert for sentence embedding derivation. nonetheless, given that sts tasks typically categorize text pairs by degrees of semantic similarity, and sentence-bert approaches these tasks from a classification standpoint, neglecting the progressive relationships between categories, there exists a clear opportunity for improvement. as an illustration, consider an sts task with five categories, labeled consecutively from 1 to 5. traditional classification strategies would yield identical loss for a sample scored at 2, irrespective of its prediction as 3 or 4, an approach evidently suboptimal. to rectify such deficiency, this paper proposes a novel framework that converts multi-class sts tasks into regression problems, thus effectively capturing the progressive relationships between categories. for a given dataset, we first map its original labels to evenly spaced numerical values, ensuring that samples with higher similarity scores are assigned correspondingly greater values. then, we set the number of nodes in the output layer to one, thereby enabling the model to produce a continuous prediction. finally, the model parameters are updated according to the difference between predicted and actual scores. distinct from standard regression tasks, the ground truth within our transformed multi-category sts tasks manifest as a series of discrete points along the numerical axis. therefore, instead of requiring precise matches to the target values, the floating-point predictions just need to be suffi-ciently close to get correctly classified. to accommodate this characteristic, we introduce a zerogradient buffer zone to widely utilized l1 loss and mse loss, unveiling two innovative loss functions: translated relu and smooth k2 loss. comprehensive evaluations across seven sts benchmarks substantiate that our regression framework surpasses traditional classification strategies in handling multi-category sts tasks. additionally, we find that our approach can further refine the performance of contrastive learning pre-trained models by utilizing filtered sts-b and sick-r training sets. these findings highlight the effectiveness of our method and underscore the importance of harnessing task-specific data, an aspect often neglected in contrastive learning paradigms. the main contributions of this study are outlined as follows: in this section, we primarily review three types of sts solutions that are directly relevant to our work: siamese neural network architectures: these approaches @xcite @xcite , proposed relatively earlier in the field, have been widely applied across various domains owing to their effectiveness on annotated corpus. although their performance on the seven sts benchmarks (sts 12-16, sts-b, sick-r) is generally inferior to contemporary contrastive learning methods, this disparity largely stems from the absence of task-specific training data. thus, models have the flexibility to opt for alternative sources, such as wikipedia @xcite or nli datasets @xcite , which adapt readily to triplet format. given our goal of tackling multi-category sts tasks, our model architecture remains rooted in the siamese network. however, in contrast to preceding efforts, we introduce an innovative regression framework specifically designed to capture the progressive relationships between categories. contrastive learning fine-tuning methods: contrastive learning is currently the mainstream paradigm for addressing sts tasks, with substantial research exploring its integration with the finetuning of plms @xcite . however, contrastive learning loss functions, epitomized by infonce loss @xcite , concentrate exclusively on binary semantic categorization and are unable to fully utilize fine-grained labeled texts. additionally, the necessity for large batch sizes to ensure negative sample diversity and prevent model collapse imposes significant computational demands. these two limitations are inherently difficult to overcome within contrastive learning itself, yet they are precisely the strengths of sentence-bert-style dual-tower models. therefore, a primary objective of this paper is to investigate whether the performance of contrastive learning models can be further enhanced by incorporating traditional siamese neural network architectures. contrastive learning pre-trained models: with the growing importance of embeddings in retrieval-augmented generation @xcite and other application scenarios, more companies and institutions are dedicating efforts to developing specialized text representation models. these approaches generally adopt multi-stage contrastive learning strategies for network pre-training @xcite @xcite . additionally, compared to large-scale generative plms, lightweight discriminative models that capture bidirectional dependencies are often more preferred. in our experiments section, we employ two state-ofthe-art contrastive learning pre-trained models, jina embeddings v2 @xcite and nomic embed @xcite . both are bert-based encoder architectures with a parameter size of 137 million. ## methodology this section presents our methodological framework, beginning with a detailed exposition of the network architecture and its operational workflow in subsection 3.1. then, in subsections 3.2 and 3.3, we introduce the two novel loss functions proposed in this study. ## experiment this section provides empirical validation of our regression framework and two innovative loss functions. we commence by comparing the performance of different modeling strategies for multicategory sts tasks and various loss functions (subsection 4.1). next, we demonstrate that, when supplemented with fine-grained training data, our siamese neural network can effectively enhance the performance of contrastive learning plms (subsection 4.2). following this, we highlight the computational efficiency of our methodology (subsection 4.3) and explore the influence of varying hyperparameter settings on model performance (subsection 4.4). finally, subsection 4.5 presents ablation studies on our network architecture. ## conclusion in this paper, we propose an innovative regression framework and develop two simple yet efficacious loss functions: translated relu and smooth k2 loss, to address multi-class sts tasks. compared to traditional classification approaches, our regression modeling strategy effectively captures the progressive relationships between categories, thereby achieving superior performance while reducing the the parameter count in the model's output layer. further empirical evidence demonstrates that our method can also be combined with leading contrastive learning models, leveraging fine-grained annotated data to further enhance their performance. moreover, this approach proves to be more advantageous than continued fine-tuning through contrastive learning, both in terms of performance gains and computational efficiency. to support further research, we have made our code and model checkpoints publicly available.