Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient (2024)

Abstract: Sentence Representation Learning (SRL) is a crucial task in Natural Language Processing (NLP), where contrastive Self-Supervised Learning (SSL) is currently a mainstream approach. However, the reasons behind its remarkable effectiveness remain unclear. Specifically, many studies have investigated the similarities between contrastive and non-contrastive SSL from a theoretical perspective. Such similarities can be verified in classification tasks, where the two approaches achieve comparable performance. But in ranking tasks (i.e., Semantic Textual Similarity (STS) in SRL), contrastive SSL significantly outperforms non-contrastive SSL. Therefore, two questions arise: First, *what commonalities enable various contrastive losses to achieve superior performance in STS?* Second, *how can we make non-contrastive SSL also effective in STS?* To address these questions, we start from the perspective of gradients and discover that four effective contrastive losses can be integrated into a unified paradigm, which depends on three components: the **Gradient Dissipation**, the **Weight**, and the **Ratio**. Then, we conduct an in-depth analysis of the roles these components play in optimization and experimentally demonstrate their significance for model performance. Finally, by adjusting these components, we enable non-contrastive SSL to achieve outstanding performance in STS.

## introduction sentence representation learning (srl) explores how to transform sentences into vectors (or "embeddings"), which contain rich semantic information and are crucial to many downstream tasks in natural language processing (nlp). in the era of large language models (llms), srl also plays an important role in providing the embedding models for retrieval-augmented generation @xcite . the quality of sentence embeddings is usually measured through transfer tasks (tr) and semantic textual similarity tasks (sts) @xcite . contrastive self-supervised learning (ssl) is now a prevalent approach in srl, which is introduced by @xcite and @xcite . it optimizes the representation space by reducing the distance between a sentence (or "anchor") and semantically similar sentences (or "positive samples"), as well as increasing the distance between the sentence and semantically dissimilar sentences (or "negative samples"). while the mechanisms underlying contrastive ssl can be intuitively understood, its effectiveness in srl has not been thoroughly explored. specifically, there are still some conflicts between the existing conclusions: (1) in theoretical perspective, machine learning community has found that contrastive ssl shares many similarities with non-contrastive ssl (e.g. alignment & uniformity @xcite , barlow twins @xcite , and vicreg @xcite ) @xcite (2) however, in practical applications, con-trastive and non-contrastive ssl show comparable performance only in classification tasks (e.g., in visual representation learning (vrl) and tr in srl @xcite ). in contrast, for ranking tasks (i.e., sts in srl), contrastive ssl seems be the only effective method, significantly outperforming non-contrastive ssl @xcite @xcite . these inconsistent conclusions suggest that to make the obtained representations suitable for sts, certain unique factors must be present in the optimization objectives, which have rarely been explored in the existing literature. in this work, we attempt to identify the key factors that enable contrastive ssl to be effective in sts. specifically, we would like to answer two questions: (1) what commonalities enable various contrastive losses to achieve superior performance in sts? (2) how can we make non-contrastive ssl, which is similar to contrastive ssl but ineffective in sts, effective? we first analyze the commonalities among four effective losses @xcite @xcite in srl from the perspective of gradients. from this analysis, we find that all gradients can be unified into the same paradigm, which is determined by three components: the gradient dissipation, the weight, and the ratio. by statistically analyzing the values of these three components under different representation space distributions, we propose three conjectures, each corresponding to the role of a component in optimizing the representation space. subsequently, we construct a baseline model to empirically validate our conjectures and demonstrate the significance of these components to model performance by varying them in the baseline. after understanding the key factors that enable contrastive losses to be effective, we are able to analyze the reasons behind the poor performance of non-contrastive ssl in sts from the perspective of three components in the paradigm. we find that these ineffective losses do not perform as well as effective ones across these components. therefore, by adjusting these components, we manage to make them function and achieve improved model performance in sts (refer to figure 1 ). briefly, our main contributions are as follows: ## conclusion in this paper, we propose a unified gradient paradigm for four different optimization objectives in srl, which is determined by three components: the gradient dissipation term, the weight term, and the ratio term. we uncover the roles these components play in optimization and demonstrate their significance to the model performance on the sts tasks. based on these insights, we succeed in making ineffective non-contrastive ssl optimization objectives effective in sts. our work advances the understanding of why contrastive ssl can be effective in srl and guides the future design of new optimization objectives.