D istill CSE : Distilled Contrastive Learning for Sentence Embeddings (2023)

Abstract: This paper proposes the DistillCSE framework, which performs contrastive learning under the self-training paradigm with knowledge distillation. The potential advantage of DistillCSE is its self-enhancing feature: using a base model to provide additional supervision signals, a stronger model may be learned through knowledge distillation. However, the vanilla DistillCSE through the standard implementation of knowledge distillation only achieves marginal improvements. The quantitative analyses demonstrate its reason that the standard knowledge distillation exhibits a relatively large variance of the teacher model’s logits due to the essence of contrastive learning. To mitigate the issue induced by high variance, this paper accordingly proposed two simple yet effective solutions for knowledge distillation: a Group-P shuffling strategy as an implicit regularization and the averaging logits from multiple teacher components. Experiments on standard benchmarks demonstrate that the proposed DistillCSE outperforms many strong baseline methods and yields a new state-of-the-art performance.

## introduction sentence embedding aims to encode the sentence's semantic information from a discrete language space to continuous dense vectors which preserves the semantic meaning of the original sentences. it plays a central role in many downstream nlp tasks, for instance, document summarization @xcite , corpus mining @xcite , and machine translation @xcite . recently, with the emergence of contrastive learning (cl), simcse @xcite pioneers the current mainstream approaches of sentence embeddings. it combines cl with pre-trained language models (plms) @xcite for training by pulling positive samples closer and pushing in-batch negatives apart, leading to state-of-the-art performance. subsequently, a plethora of sentence embedding methods has been developed based on the simcse framework @xcite @xcite @xcite @xcite @xcite @xcite . unfortunately, one problem with contrastive learning for sentence embeddings is that the construction of positive and negative sample pairs is often too simple, making it easy for the model to distinguish between positive and negative pairs. as a result, the model may not learn very informative knowledge, leading to sub-optimal performance @xcite . to this end, inspired by self-training @xcite , this paper proposes a framework -distillcse-which performs contrastive learning under the self-training paradigm with knowledge distillation @xcite . the advantage of distillcse is its self-enhancing feature: using a base model to provide additional supervision signals, a stronger model can be learned through knowledge distillation. specifically, our framework can be divided into three steps ( §2): first, it learns a base model as a teacher using standard contrastive learning; second, it learn a stronger student model through knowledge distillation @xcite ; thrid, it iteratively repeats the process of knowledge distillation by treating the student model as a teacher. however, it is far from easy to put distillcse into practice: our preliminary experiment shows that the vanilla implementation of the proposed framework only achieves marginal improvements. we identify that the vanilla distillation method suffers from the severe overfitting on the training corpus (see in summary, our contribution is three-fold: ## distillcse for sentence embeddings self-training utilizes a trained model to generate synthetic labels for examples and trains a student model from scratch @xcite . this approach is can be implemented by knowledge distillation @xcite when the student model has a equal or smaller capacity than the teacher model. self-training via knowledge distillation involves two main steps in the context of sentence embeddings as follows. step 1: training teacher model in the initial phase, a teacher model is trained for sentence embeddings. the mainstream method for sentence embeddings is simcse @xcite , which incorporates contrastive learning (cl) principles into the learning process. simcse maximizes the agreement of samples with its positive instances while pushing away all the in-batch negative examples using the cl loss defined as follows: step 2: distilling the student model for the second phase, the student model learns from scratch to mimic the output of the teachers. in this paper, we specifically focus on using a homogeneous model structure for both the teacher and student models (i.e. same capacity). to achieve this, we utilize the vanilla knowledge distillation loss @xcite , which aims to minimize the cross-entropy of the output logits between the teacher and student. formally, given a sentence x i and its corresponding set of in-batch sentence pairs {(x i , x j )} n @xmath0,j̸ =i , distillation minimizes the cross entropy loss between the teacher distribution q and the student distribution p according to the following objective: here, s i,j and t i,j represent the cosine similarity logits of sentence pairs calculated by the student and teacher models, respectively. τ s and τ t denote the corresponding temperatures for the student and teacher models, and n denotes the number of sentences within a mini-batch. in general, the student model is commonly trained jointly with the teacher model's training objective. hence, the objective for self-training on sentence embeddings combines both objectives using a trade-off factor λ: iterative self-training since single-round selftraining enhances the student model performance beyond the baseline teacher, a natural assumption is to employ the student as the teacher for the next round of distillation. consequently, the next-round student could be further improved, which is widely supported by almost all existing findings @xcite . more specifically, iterative self-training employs the r-th round student as the r + 1-th round teacher: vanilla distillation tends to overfitting however, our preliminary experiments on vanilla distillation does not obtain significant improvements, which is in line with the finding in @xcite . in particular, our further analysis shows that there is a large gap between the loss of the student model on the training and testing sets (see ## high variance issues and solutions in this section, we will first point out the high variance from the logits that causes the overfitting for knowledge distillation in contrastive learning (i.e., step 2 in distillcse framework) and then propose two simple yet effective solutions to mitigate such issues. ## conclusion in this paper, we propose a self-training with the knowledge distillation framework for contrastive sentence embeddings. we identify that vanilla distillation suffers from severe overfitting issue. the reason for this problem lies in the significant variance of the output logits of the base model in selftraining, both among data points and across teachers. furthermore, reducing the variance will lead to better student performance. consequently, we propose group-p shuffling to regulate the variance and mean sampling from multiple teacher components to reduce the logit variance. experimental results on standard benchmarks demonstrate the effectiveness of our proposed method, which yields a new state-of-the-art performance.