TSDAE : Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning (2021)

Abstract: Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming other approaches like Masked Language Model. A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.

## introduction sentence embedding techniques encode sentences into a fixed-sized, dense vector space such that semantically similar sentences are close. the most successful previous approaches like infersent @xcite , universial sentence encoder (use) @xcite and sbert @xcite heavily relied on labeled data to train sentence embedding models. however, for most tasks and domains, labeled data is not available and data annotation is expensive. to overcome this limitation, unsupervised approaches have been proposed which learn to embed sentences just using an unlabeled corpus for training. we propose a new approach: transformer-based sequential denoising auto-encoder (tsdae). it significantly outperforms previous methods via an encoder-decoder architecture. during training, ts-dae encodes corrupted sentences into fixed-sized vectors and requires the decoder to reconstruct the original sentences from this sentence embedding. for good reconstruction quality, the semantics must be captured well in the sentence embedding from the encoder. later, at inference, we only use the encoder for creating sentence embeddings. a crucial shortcoming of previous unsupervised approaches is the evaluation. often, approaches are mainly evaluated on the semantic textual similarity (sts) task from semeval @xcite @xcite . as we argue in section 4, we perceive this as an insufficient evaluation. the sts datasets do not include sentences with domain specific knowledge, i.e., it remains unclear how methods will perform on more specific domains. further, sts datasets have an artificial score distribution, and the performance on sts datasets does not correlate with downstream task performances @xcite . in conclusion, it remains unclear, how well unsupervised sentence embedding methods will perform on domain specific tasks. to answer this question, we compare tsdae with previous unsupervised sentence embedding approaches on three different tasks (information retrieval, re-ranking and paraphrase identification), for heterogeneous domains and different text styles. we show that tsdae can outperform other state-of-the-art unsupervised approaches by up to 6.4 points. tsdae is able to perform on-par or even outperform existent supervised models like use-large, which had been trained with a lot of labeled data from various datasets. further, we demonstrate that tsdae works well for domain adaptation and as a pre-training task. we observe a significant performance improvement compared to other pre-training tasks like masked language model (mlm). our contributions are three-fold: unsupervised sentence embeddings utilize only an unlabeled corpus during training. recent work combined pre-trained transformers with different training objectives to achieve state-of-the-art results on sts tasks. among them, contrastive tension (ct) @xcite simply views the identical and different sentences as positive and negative examples, resp. and train two independent encoders; bert-flow @xcite trains model via debiasing embedding distribution towards gaussian; simcse @xcite is based on contrastive learning @xcite and views the identical sentences with different dropout mask as the positive examples. for more details, please refer to section 5. all of them pooling encoder decoder text without noise text with noise requires only independent sentences. by contrast, declutr @xcite utilizes sentencelevel contexts and requires long documents (2048 tokens at least) for training. this requirement is hardly met for many cases, e.g. tweets or dialogues. thus, in this work we only consider methods which uses only single sentences during training. most previous work mainly evaluate only on semantic textual similarity (sts) from the semeval shared tasks. as we show in section 4, the unsupervised approaches perform much worse than the out-of-the-box supervised pre-trained models even though they were not specifically trained for sts. further, a good performance on sts does not necessarily correlate with the performance on downstream tasks @xcite . it remains unclear how these methods perform on specific tasks and domains. to answer this, we compare three recent powerful unsupervised methods based on pre-trained transformers including ct, simcse, bert-flow and our proposed tsdae on different tasks of heterogeneous domains. ## sequential denoising auto-encoder although sequential denoising auto-encoder (sdae) @xcite @xcite is a popular unsupervised method in machine learning, how to combine it with pre-trained transformers for learning sentence embeddings remains unclear. in this section, we first introduce the training objective of tsdae and then give the optimal configuration of tsdae. ## evaluation previous unsupervised sentence embedding learning approaches @xcite @xcite @xcite primarily evaluated on the task of semantic textual similarity (sts) with data from semeval using pearson or spearman's rank correlation. we find the (sole) evaluation on sts problematic. as shown in @xcite , performance on the sts dataset does not correlate with downstream task performance, i.e. an approach working well on the sts tasks must not be a good choice for downstream tasks. we confirm this with our experiments, the performance on the sts tasks does not correlate with the performance on other (real-world) tasks. see section 6.1 for more details on this. this has multiple reasons: first, the sts datasets consists of sentences which do not require domainspecific knowledge, they are primarily from news and image captions. it is unclear how approaches will work for domain-specific tasks. second, the sts datasets have an artificial score distributiondissimilar and similar pairs appear roughly equally. for most real-word tasks, there is an extreme skew and only a tiny fraction of pairs are considered similar. third, to perform well on the sts datasets, a method must rank dissimilar pairs and similar pairs equally well. in contrast, most real-world tasks, like duplicate questions detection, related paper finding, or paraphrase mining, only require to identify the few similar pairs out of a pool of millions of irrelevant combinations. a further shortcoming of previous evaluation setups is just testing the case of unsupervised learning, ignoring labeled data that potentially exists. in many scenarios, some labeled data exists, either directly from the specific task or from other (similar) tasks. a good approach should also work if some labeled data is available. hence, we propose to evaluate unsupervised sentence embedding approaches in following three setups: unsupervised learning: we assume we just have unlabeled sentences from the target task and tune our approaches based on these sentences. ## experiments in this section, we compare our proposed tsdae with other unsupervised counterparts and out-ofthe-box supervised pre-trained models on the above mentioned tasks. for comparison, we include three recent state-of-the-art unsupervised approaches: ct, simcse, and bert-flow. we use the proposed hyper-parameters from the respective paper. without other specification, bert-base-uncased foot_2 is used as the base transformer model. to eliminate the influence of randomness, we report the scores averaged over 5 random seeds. for other details, please refer to appendix b. ## analysis we analyze how many training sentences are needed and if relevant content words are identified. for all the datasets except twitterpara, the analysis is carried out on the development set. for twitterpara, the test set is used, as it has no development split released by the original paper. all the hyper-parameters are chosen up-front without tuning to a particular dataset. ## discussion we mainly experiment with pre-trained transformer encoders in this work. besides single encoders, there are also pre-trained encoder-decoder models like bart @xcite and t5 @xcite . however, they are already extensively pre-trained with variants of auto-encoder loss on the general domain and they are suspected of overfitting the reconstruction behavior. to verify this idea, we also further train bart-base and t5-base models with tsdae on the 4 domainspecific datasets. the results are shown in ## conclusion in this work, we propose a new unsupervised sentence embedding learning method based on pretrained transformers and sequential deoising autoencoder (tsdae). we evaluate tsdae on other, recent state-of-the-art unsupervised learning on four different tasks from heterogeneous (specialized) domains in three different settings: unsupervised learning, domain adaptation, and pretraining. we observe that tsdae performs well on the selected tasks and for the different settings, significantly outperforming other approaches. further, we show that the current evaluation of unsupervised sentence embedding learning approach, which is primarily done on the semantic textual similarity (sts) task, is insufficient: a strong performance on sts does not correlate with a good performance on specific tasks. many recent unsupervised approaches are not able to outperform out-of-the-box pre-trained models on the selected tasks.