Improving Contrastive Learning of Sentence Embeddings with Focal I nfo NCE (2023)

Abstract: The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives. Experimentation on various STS benchmarks shows that our method improves sentence embeddings in terms of Spearman’s correlation and representation alignment and uniformity.

## introduction unsupervised learning of sentence embeddings has been extensively explored in natural language processing (nlp) @xcite @xcite , aiming to generate meaningful representations of sentences without the need for labeled data. among various approaches, sim-cse @xcite achieves state-of-the-art performance in learning high-quality sentence embeddings through contrastive learning. due to its simplicity and effectiveness, various efforts have been made to improve the contrastive learning of sentence embeddings from different aspects, including alleviating false negative pairs @xcite and incorporating more informative data augmentations @xcite . leveraging hard-negative samples in contrastive learning is of significance @xcite @xcite . nevertheless, unsupervised contrastive learning ap-proaches often face challenges in hard sample mining. specifically, the original training paradigm of unsupervised-simcse proposes to use contradiction sentences as "negatives". but such implementation only guarantees that the contradiction sentences are "true negatives" but not necessarily hard. with a large number of easy negative samples, the contribution of hard negatives is thus prone to being overwhelmed, to address this issue, we propose a novel loss function, namely focal-infonce, in the paradigm of unsupervised simces for sentence embedding. inspired by the focal loss @xcite , the proposed focal-infonce loss assigns higher weights to the harder negative samples in model training and reduces the influence of easy negatives accordingly. by doing so, focal info-nce encourages the model to focus more on challenging pairs, forcing it to learn more discriminative sentence representations. in addition, to adapt the dropout strategy for positive pair construction in simcse, we further incorporate a positive modulation term in the contrastive objective, which reweights the positive pairs in model optimization. we conduct extensive experiments on various sts benchmark datasets @xcite @xcite @xcite @xcite to evaluate the effectiveness of focal info-nce. our results demonstrate that focal info-nce significantly improves the quality of sentence embeddings and outperforms unsupervised-simcse by an average of 1.64%, 0.82%, 1.51%, and 0.75% spearan's correlation on bert-base, bert-large, robertabase, and roberta-large, respectively. ## focal-infonce for sentence embedding this study follows the unsupervised simcse framework for sentence embedding. instead of taking the infonce loss in eq. ( 1 ), we introduce a self-paced reweighting objective function, focal-infonce, to up-weight hard negative samples in contrastive learning. specifically, for each sentence x i , focal-infonce is formulated as analysis of focal-infonce: compare with in-fonce in eq. ( 1 ), focal-infonce introduces selfpaced modulation terms on s p and s n , proportional to the similarity quantification. let's first focus on the modulation term, s i,j n + m, on negative pairs. prior arts have shown that pre-trained language models usually suffer from anisotropy in sentence embedding @xcite . finetuning the pretrained models with contrastive learning on negative samples, especially hard negative samples, improves uniformity of representations, mitigating the anisotropy issue. in simcse, s i,j n quantifies the similarity between negatives x i and x j . if s i,j n is large, x i and x j are hard negatives for current model. improving the model with such hard negative pairs encourage representation's uniformity. to this end, we propose to upweight the corresponding term s i,j n /τ by a modulation factor s i,j n + m. the partial derivative of focal-infornce with respect to s i,j n is according to eq. ( 4 ), comparing to easy negatives, hard negative samples that associates with higher similarity score s i,j n contribute more to the loss function. this implies that a model optimized with the proposed focal-infonce focuses more on hardnegative samples. our experiments also show that focal-infonce improves uniformity in sentence embeddings. to uncover the insight of the modulation term s i p on positive cases, let's revisit simcse. in sim-cse, the positive pair is formed by dropout with random masking. thus a low similarity score s p indicates semantic information loss introduced by dropout. since such a low similarity is not attributed to model's representation capability, we should mitigate its effect on model optimization. hence, focal-infornce assigns a small weight to the dissimilary positive pair. the partial derivative with respect to s i,j n is due to the modulation terms on both positive and negative samples, focal-infonce reduces the chances of the model getting stuck in sub-optimal solutions dominated by easy pairs. we show in our experiment that the proposed focal-infonce can easily fit into most contrastive training frameworks for sentence embeddings. ## experiments we evaluate focal-infonce on 7 semantic similarity tasks: sts 12-16 @xcite @xcite @xcite , sts benchmark @xcite and sick-relatedness @xcite . spearman's correlation between predicted and ground truth scores is used as the numerical performance metric. our implementation closely follows unsupervised-simcse @xcite . briefly, starting with pre-trained models bert base , bert large @xcite and roberta base , and roberta large @xcite , we take the [cls] embeddings as the sentence representations and fine-tune the models with 10 6 randomly sampled english wikipedia sentences. the hyperparameter τ for the four models are {0.7, 0.7, 0.5, 0.5} respectively and we use @xmath0.3 in this experiment. to make a fair comparison, we adopted the same batch size and learning rate as unsupervised-simcse, shown in ## conclusions this paper introduced a novel unsupervised contrastive learning objective function, focal-infonce, to enhance the quality of sentence embeddings. by combining simcse with self-paced hard negative re-weighting, model optimization was benefited from hard negatives. extensive experiments shows the effectiveness of the proposed method on various sts benchmarks.