S im CSE ++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives (2023)

Abstract: This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model’s performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.

## introduction sentence representation, which transforms sentence semantic information from discrete language space into dense vectors, is one of the most fundamental tasks in natural language processing, as it serves as the central role for a wide range of downstream applications (e.g., information retrieval, semantic comparison, question answering, and language translation). sentence representation has been constantly evolving @xcite @xcite , and it achieves even stronger performance when utilizing pre-trained language models (plm) @xcite . moreover, on top of plms, a number of post-processing strategies achieve even better performance. for example, @xcite employs a flow-based model and @xcite applies the whitening process to flatten a uniform distribution of representations. more recently, remarkable advancements have been achieved by contrastive learning (cl) on sentence embeddings @xcite , which cleverly makes use of dropout randomness @xcite to construct positive pairs in an unsupervised way. since then, many notable variants have been proposed under the contrastive learning framework to intensify performance by constructing hard contrastive pairs @xcite @xcite @xcite , introducing other cl-based objectives @xcite @xcite @xcite or utilizing more sophisticated similarity metrics @xcite . in this paper, we improve the sentence embedding models from two perspectives: dropout noise and feature corruption. specifically, first, we empirically study the effects of dropout randomness on positive pairs and negative pairs in the cl-based objective. we find that modest dropout noise in the positive pairs is beneficial to the model performance whereas dropout noise in negative pairs is harmful. we provide an explanation from the principle of noise contrastive estimation @xcite and the role of dropout in constructing positive pairs. based on these findings, we propose a simple yet effective strategy, offdropout, which turns off the dropout randomness in negative pairs to further improve the performance. second, we revisit the issue of feature corruption on the sentence embedding and empirically study the well-known solution recently proposed by @xcite ; @xcite to this problem. surprisingly, we find that this solution does not improve performance under the contrastive learning framework for sentence embeddings. we further analyze this finding and identify the reason behind it as the rank bottleneck issue in the mini-batch embedding matrix. to tackle this issue, we propose a simple dimension-wise contrastive learning (dcl) to break down the bottleneck, which eventually enhances the baseline performance. as a result, by combining the proposed offdropout and dcl, we have advanced the simcse baseline by 1.9 points. furthermore, our reproduced results have shown that we advanced the current state-of-the-art model, diffcse @xcite , by 1.4 points. in general, our contribution is three-fold: ## improving dropout noise in cl simcse framework plays a central role in recent sentence embedding strategies. it is a simple contrastive learning framework that learns by identifying positive pairs among in-batch negatives. specifically, for a given sentence x i , let f (•) denotes a pre-trained language model, and it is used to generate two views (z foot_0 i , z 2 i ) of the identical sentences x i via different dropout patterns: simcse @xcite aims to maximize the agreement between positive pairs z 1 i , z 2 i and minimize the n -1 in-batch negatives z 1 i , z 2 j using the infonce objective @xcite : here, s(•, •) is the similarity measure between two inputs (i.e., cos_sim(•, •)/τ , where τ is the temperature). in equation ( 2 ), z 1 i , z 2 j is a negative pair, and the dropout random variable ξ is used as an augmentation function for positive pairs, i.e., z 1 i , z 2 i . ## mitigating feature corruption feature corruption issue feature corruption foot_1 (chen and he, 2021) illustrates the issue that each dimension of the output representation has high similarity with the other dimensions. such correlation between dimensions reduces the model's representation capability and undermines downstream performance @xcite . @xcite proposes barlowtwins as an additive regulation to tackle such an issue, which is a dimension decorrelation objective. bar-lowtwins tackles feature corruption by minimizing the redundancy between each dimension and aims to produce dimensional-independent representations. formally, given a cross-correlation matrix @xmath0, its objective is: ## conclusion in this paper, we improve cl-based sentence embeddings in dropout noise and feature corruption. the main findings are: 1) having modest dropout noise is successful for positive pairs and reducing dropout noise from positive pairs is harmful whereas reducing dropout noise from negative pairs is beneficial; 2) the well-known solution to feature corruption does not lead to gains on sentence embedding due to the rank bottleneck issue. accordingly, we propose off-dropout to eliminate the dropout randomness from negative pairs and dimension-wise cl objective to break the bottleneck to alleviate feature corruption, both of which outperform strong baselines by a margin.