An Efficient Self-Supervised Cross-View Training For Sentence Embedding (2023)

Abstract: Self-supervised sentence representation learning is the task of constructing an embedding space for sentences without relying on human annotation efforts. One straightforward approach is to finetune a pretrained language model (PLM) with a representation learning method such as contrastive learning. While this approach achieves impressive performance on larger PLMs, the performance rapidly degrades as the number of parameters decreases. In this paper, we propose a framework called Self-supervised Cross-View Training (SCT) to narrow the performance gap between large and small PLMs. To evaluate the effectiveness of SCT, we compare it to 5 baseline and state-of-the-art competitors on seven Semantic Textual Similarity (STS) benchmarks using 5 PLMs with the number of parameters ranging from 4M to 340M. The experimental results show that STC outperforms the competitors for PLMs with less than 100M parameters in 18 of 21 cases.1

## introduction self-supervised sentence representation learning is the task of constructing an embedding space for sentences without relying on human annotation efforts. recent advancements in self-supervised sentence representation present promising results on various downstream tasks such as semantic textual similarity (sts) and text classification. for example, @xcite found that self-supervised sentence embedding methods could be on par with supervised methods @xcite on various sts benchmarks. a straightforward approach to self-supervised sentence representation is to finetune a pre-trained language model (plm), i.e., bert @xcite and roberta @xcite , with a representation learning technique. one popular method is contrastive learning. this learning method enables self-supervised representation learning by creating a self-referencing mechanism through data augmentation @xcite @xcite @xcite @xcite . these works have demonstrated improvements over existing self-supervised techniques in sentence embedding benchmark datasets (i.e., sts and text classification). figure 2 shows how three existing methods-sim-cse @xcite , diffcse @xcite , and dclr @xcite perform on the bert architecture as we varied the number of parameters from 4m to 340m. while these self-supervised techniques achieve impressive performance on larger plms (i.e., those with more than 100m parameters), the performance rapidly degrades as the number of parameters decreases @xcite @xcite . the figure also shows how the data points organize themselves into two distinct groups: ll and hh. despite the apparent benefit of low computation costs, smaller models, i.e., bert-tiny, bert-mini, and bert-small, are often neglected. greater emphasis should be placed on exploring the potential to enhance the performance of smaller models through novel learning methods specifically tailored to their unique characteristics. in this paper, we propose a framework called self-supervised @xcite to narrow the performance gaps between large and small plms. figure 1 displays the difference between the traditional contrastive learning approach and ours. as shown in figure 1a , the two views are separated for contrastive learning, and the outputs from h(•) are directly compared to each other. figure 1b highlights the key distinctions of sct based on two concepts: cross-view comparison and similarity-score-distribution learning. the combination of these two concepts provides additional guidance which improves the effectiveness of self-supervised sentence representation learning on small plms. to evaluate the effectiveness of sct, we compare it to state-of-the-art (sota) competitors on sts, re-ranking, and natural language inference (nli) benchmarks. we also employ a distillation setting using bert-large-simcse as a teacher model. the experimental results on sts demonstrated that our framework could address the drastic performance degradation problems in small plms by outperforming competitors in every case when the number of parameters is less than 100m. for the smallest model (#parameter: 4 million), we improved the performance from 64.47 to 69.73 points compared to simcse. in the case of large plms (i.e., those with more than 100m parameters), our model's performance was on par with the current sota model when tested on bert-base and bert-large. for the distillation setting, we outperformed all distillation competitors on all plms. for the re-ranking and nli tasks, we improved the downstream tasks' performance for nearly all settings. the contributions of our work are as follows: ## related work self-supervised learning is becoming more popular as a method to learn sentence representation from pre-trained language models (plms) without annotated information from training corpora. we cover well-known self-supervised sentence representation learning techniques in the following subsections. ## proposed method one of the challenges of using small models is the limited number of parameters. an empirical study has shown that larger models have enough parameters to solve complex problems with simple techniques, while smaller ones require more guidance to solve complex problems @xcite . based on this observation, we design our proposed solution, self-supervised cross-view training (sct), to enhance the learning guidance for smaller models (those with less than 100m parameters) by improving the self-referencing and discrepancy measurement mechanisms. figure 3 illustrates the sct pipeline and highlights the two mechanisms we introduce to improve the learning guidance: cross-view comparison pipeline and similarity-score-distribution learning. in what follows, we describe how the cross-view pipeline improves the robustness of the self-referencing mechanism in section 3.1. section 3.2 presents the mechanism we use to measure the discrepancies between cross-view outputs. we explain our proposed sct loss function in section 3.3. finally, we introduce sentence representation distillation into our proposed framework in section 3.4. ## experimental results this section presents results from five sets of studies. section 5.1 presents results from the ## conclusion we propose a self-supervised sentence representation learning method called self-supervised cross-view training (sct). the observation inspires our work that smaller models, when constructed in a self-supervised setting, tend to perform poorly or collapse altogether. we hypothesize that this problem can be addressed by providing additional learning guidance to facilitate the self-referencing mechanism in the self-supervised learning pipeline. our work consists of three key contributions. first, at the framework level, we formulate a cross-view comparison pipeline to improve the self-referencing mechanism by enabling crosscomparison between two input views. in addition, our framework allows using two input views formulated from the same or different plms. second, to facilitate the learning process, we also design a new technique to measure the discrepancy between two cross-view outputs. instead of comparing them directly, we use similarity score distributions. third, we conducted extensive sets of experimental studies to compare our method against existing competitors and to analyze our design decisions. the experimental results on the sts tasks show that our method dominates all competitors in the cases of plms with less than 100m parameters. with the help of the distillation loss, our method improves the performance of small plms better than that of large plms. moreover, our method outperforms competitive methods for all plms on the downstream tasks. furthermore, the results also confirm that the cross-view comparison pipeline and similarity score distribution comparison are crucial to performance improvement. these findings imply that smaller plms benefit from our judiciously designed guidance in a self-supervised setting.