Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals (2022)

Abstract: There have been many successful applications of sentence embedding methods. However, it has not been well understood what properties are captured in the resulting sentence embeddings depending on the supervision signals. In this paper, we focus on two types of sentence embedding methods with similar architectures and tasks: one fine-tunes pre-trained language models on the natural language inference task, and the other fine-tunes pre-trained language models on word prediction task from its definition sentence, and investigate their properties. Specifically, we compare their performances on semantic textual similarity (STS) tasks using STS datasets partitioned from two perspectives: 1) sentence source and 2) superficial similarity of the sentence pairs, and compare their performances on the downstream and probing tasks. Furthermore, we attempt to combine the two methods and demonstrate that combining the two methods yields substantially better performance than the respective methods on unsupervised STS tasks and downstream tasks.

## introduction sentence embeddings are dense vector representations of a sentence. a variety of methods have been proposed to derive sentence embeddings, including those based on unsupervised learning @xcite @xcite @xcite and supervised learning @xcite . pre-trained transformer-based @xcite language models, such as bert @xcite and roberta @xcite , have been successfully applied in a wide range of nlp tasks, and sentence embedding methods that leverage pre-trained language models have also performed well on semantic textual similarity (sts) tasks and several downstream tasks. these methods refine pre-trained language models for sophisticated sentence embeddings by unsupervised learning @xcite wang and kuo, 2020; @xcite @xcite , or supervised learning @xcite @xcite . among them, @xcite proposed sentence-bert (sbert), which fine-tunes pre-trained language models on the natural language inference (nli) task. sbert performed well on the sts and downstream tasks. recently, @xcite proposed defsent, which fine-tunes pre-trained language models on the task of predicting a word from its definition sentence in a dictionary, and reported that it performed comparably to sbert. figure 1 shows overviews of sbert and defsent. although both methods finetune the same pre-trained models and use the same pooling operations to derive a sentence embedding, the supervision signals for fine-tuning are different. that is, sbert leverages nli datasets, whereas defsent leverages word dictionaries. it is expected that the properties of the sentence embeddings depend on their supervision signals. however, since existing research has mainly focused on achieving better performance on benchmark tasks, it has not been revealed what property differences the resulting sentence embeddings have. investigating the properties of sentence embeddings would give us a better understanding of existing sentence embedding methods and help develop further methods. in this paper, we empirically investigate the influence of supervision signals on sentence embeddings. we focus on sbert and defsent because they leverage different supervision signals but have very similar architectures, as shown in figure 1 ; thus, they would be appropriate for analyzing the influence of the supervision signals on sentence embeddings. first, we partitioned the sts datasets @xcite @xcite @xcite @xcite on the basis of two different perspectives and examine what type of meaning each type of sentence embeddings captures by analyzing the performance of each method on these partitioned sts datasets. we then apply each type of embeddings to the downstream and probing tasks of senteval @xcite and analyze what type of information is captured. our results demonstrate that the supervision signals have a significant impact on performance on these tasks and that the properties of sbert and defsent would be complementary. thus, we further explore whether combining the two methods yields better sentence embeddings to confirm their complementarity, and demonstrate that combining the two methods yields substantially better performance than the respective methods on unsupervised sts tasks and downstream tasks of senteval. ## preparation in this section, we present detailed descriptions of sbert and defsent, the two sentence embedding methods compared in this study, and describe the tasks and settings for the experiments. ## comparison of sentence embeddings the supervision signal used for fine-tuning sentence embeddings might affect their properties. for example, since it is crucial to capture the differences in meaning even when the given sentence pair is superficially similar in the nli task, sbert is considered suitable for determining the semantic similarity between superficially similar sentence pairs. in this section, we attempt to reveal such properties of each type of sentence embeddings. first, we partition the sts datasets on the basis of the source of the sentence pairs and the superficial similarity of the sentence pair. we then apply each type of embeddings to the downstream and probing tasks of senteval. ## combination of sentence embeddings we have shown that sbert and defsent have different properties and that they may be complimentary. this suggests that combining the two methods may yield better sentence embeddings. thus, we attempt to combine sbert and defsent and evaluate the resulting sentence embeddings on unsupervised sts tasks and senteval downstream tasks. specifically, we use the following five methods of combining sbert and defsent for bert foot_1 . s+d fine-tuning the pre-trained model with sbert then with defsent sequentially. d+s fine-tuning the pre-trained model with def-sent then with sbert sequentially. ## related work sentence embedding has been studied intensively. @xcite proposed skipthought, which trains a sentence embedding model by predicting the previous and next sentence from the embedding of a given sentence. @xcite proposed infersent, which trains a sentence embedding model built on bilstm in a siamese network architecture on the nli task. @xcite proposed universal sentence encoder (use), which is trained on an nli dataset, and has also shown the effectiveness of nli datasets in obtaining sophisticated sentence embeddings. recently, methods that leverage pre-trained language models to acquire sentence embeddings have attracted much attention. pre-trained language models, such as bert @xcite and roberta @xcite knowledge by training on large texts and perform well on downstream tasks. pre-trained models are also considered helpful for sentence embedding. there are two types of methods based on pre-trained models: unsupervised and supervised. unsupervised methods do not require labeled text but exploit the properties of pre-trained language models or create training data artificially. @xcite showed that the sentence embedding space of bert is anisotropic, and proposed bertflow, which learns a map to an isotropic gaussian distribution to obtain sentence embedding. several studies have also been based on contrastive learning, and are different in the way to make positive examples: declutr @xcite takes into account different spans of the same document as positives; consert @xcite takes into account a pair of an original sentence and a collapsed sentence as positives; unsupervised simcse @xcite takes into account the corresponding embeddings of the same sentence with different dropout masks applied as positives. supervised methods use labeled text to encode higher-level semantic information. supervised methods generally produce more sophisticated sentence embeddings than unsupervised methods. in addition to sbert and defsent, supervised sim-cse @xcite is one of the supervised sentence embedding methods. supervised simcse fine-tunes bert by contrastive learning using entailment pairs in the nli datasets as positives. ## conclusion in this paper, we empirically investigated the influence of supervision signals used for obtaining sentence embeddings. we focused on two methods: sbert, which uses nli datasets, and defsent, which uses word dictionaries. we showed that there is a difference in the ability to order the similarity of sentences depending on their source or superficial similarity by comparing their performances on subsets of the sts datasets and tasks of senteval. we found that sbert is suitable for superficially similar sentence pairs because sbert is based on the nli datasets that contain a relatively large number of superficially similar sentences, whereas defsent is suitable for sentence pairs that need to represent the compositional meaning because defsent is based on definition sentences of a dictionary. we also showed that sbert performed better in tasks where sentiment information was important, while defsent performed better in tasks where information about words and the compositionality of meaning were important by comparing their performances on downstream and probing tasks of senteval. finally, we demonstrated that combining the two methods yielded substantially better performance than the respective methods on unsupervised sts tasks and downstream tasks of senteval. for future work, we will expand the scope of our analysis to other pre-trained language models and sentence embedding methods to obtain insights for better sentence embeddings. in addition, we will investigate how those combination methods affect the properties of resulting sentence embeddings and explore how to effectively combine unsupervised sentence embedding methods, which have recently achieved good performance, such as declutr @xcite and unsupervised simcse @xcite , with supervised sentenece embedding methods. moreover, the combination of unsupervised methods, which have recently achieved good performance, such as declutr @xcite and unsupervised simcse @xcite , and supervised methods should also be promising.