S ent B ench: Comprehensive Evaluation of Self-Supervised Sentence Representation with Benchmark Construction (2023)

Abstract: “Self-supervised learning has been widely used to learn effective sentence representations. Previ-ous evaluation of sentence representations mainly focuses on the limited combination of tasks andparadigms while failing to evaluate their effectiveness in a wider range of application scenarios. Such divergences prevent us from understanding the limitations of current sentence representa-tions, as well as the connections between learning approaches and downstream applications. Inthis paper, we propose SentBench, a new comprehensive benchmark to evaluate sentence repre-sentations. SentBench covers 12 kinds of tasks and evaluates sentence representations with threetypes of different downstream application paradigms. Based on SentBench, we re-evaluate sev-eral frequently used self-supervised sentence representation learning approaches. Experimentsshow that SentBench can effectively evaluate sentence representations from multiple perspec-tives, and the performance on SentBench leads to some novel findings which enlighten futureresearches.”

## introduction self-supervised representation learning is considered an important reason for breakthroughs in nlp @xcite @xcite . and learning effective sentence representations has long been a fundamental challenge. @xcite @xcite . in recent years, various self-supervised sentence representation learning approaches leverage different self-constrained signals, e.g., sentence pairs in the same narratives @xcite , sentence order @xcite , or sentence permutation @xcite , to learn representations by training models to distinguish positive instances from negatives. even though current self-supervised sentence representation approaches have reached significant progress on some datasets like semantic textual similarity (sts) @xcite , benchmarks for evaluation lag far behind the development of methods @xcite . currently, sentence representations are evaluated in limited tasks and specific paradigms. for example, the most commonly used senteval benchmark @xcite mainly focuses on single sentence classification and semantic similarity tasks. unfortunately, prior literature shows that performance on sts cannot reflect the effectiveness of sentence representations on a wider range of tasks @xcite @xcite . and available evaluation toolkits assess the same downstream task with a singular paradigm, limiting our perception of methods in different application scenarios. moreover, current self-supervised sentence representation learning approaches are coupled with multiple factors, including diverse contrastive signals, training losses, and model architectures. consequently, evaluating whether, where, and how a learning method will benefit the downstream tasks is difficult. in this paper, we propose sentbench, a new benchmark to comprehensively evaluate sentence representations with various downstream tasks and evaluation paradigms. as shown in figure 1 , sentbench contains 12 kinds of nlp tasks, including sentiment classification, question answering, story cloze, etc., and three evaluation paradigms, including single sentence classification, sentence pair classification and sentence pair contrasting @xcite . the classification paradigm trains a simple additional classifier to assess information within representations for single sentence tasks or identify the connection between two candidate representations for pair-wise tasks. besides, contrasting paradigm is similar to common retrieval or ranking scenario. finally, sentbench constructs 18 datasets, which cover diverse tasks and common applications of sentence representations. based on sentbench, we re-evaluate several widely used self-supervised sentence representation learning approaches. we decouple previous approaches from two perspectives to identify critical factors: contrasting knowledge applied to construct positive instances and training losses used to optimize models. specifically, we concentrate on three contrasting knowledge, including next sentence prediction @xcite , self-contrasting @xcite and data augmentation @xcite , as well as two widespread training losses, including contrastive loss and classification loss. by thoroughly comparing different approaches on sentbench, we find that the advantages of the state-of-the-art methods can not be exhibited consistently to a broader range of downstream tasks and evaluation paradigms. furthermore, the applied training loss leads to more significant impacts than contrasting knowledge. these findings shed some light on future research on sentence representation learning. single sentence tasks aim to classify sentence representations into corresponding categories. because the previous senteval 0 benchmark has covered extensive single sentence classification tasks, sentbench inherits all of them, including sentiment analysis (mr, sst) @xcite , opinion polarity (mpqa, subj) @xcite ), question type (trec) (voorhees and tice, 2000), product reviews (cr) @xcite . sentence pair tasks aim to identify sentence pairs with specific connections. we investigate six tasks covering various fields of downstream applications of nlp ( ## experiment setup based on sentbench, we re-evaluate several most frequently used self-supervised sentence representation methods. since contrasting knowledge and training losses are usually coupled, it is challenging to directly identify critical factors for successful sentence representations from previous works. to this end, this paper explores different combinations of contrasting knowledge and training losses to investigate the effects of distinct factors. ## conclusion in this paper, we propose a new universal sentence evaluation benchmark sentbench, which introduces more downstream tasks and evaluation paradigms. furthermore, we decouple and analyze the effects of contrasting knowledge and training losses on sentence representations. empirical findings show that training losses play a more critical role in self-supervised sentence representation learning and help us better understand and design sentence representation learning algorithms.