W hitening BERT : An Easy Unsupervised Sentence Embedding Approach (2021)

Abstract: Producing the embedding of a sentence in anunsupervised way is valuable to natural language matching and retrieval problems in practice. In this work, we conduct a thorough examination of pretrained model based unsupervised sentence embeddings. We study on fourpretrained models and conduct massive experiments on seven datasets regarding sentence semantics. We have three main findings. First, averaging all tokens is better than only using [CLS] vector. Second, combining both topand bottom layers is better than only using toplayers. Lastly, an easy whitening-based vector normalization strategy with less than 10 linesof code consistently boosts the performance. The whole project including codes and data is publicly available at https://github.com/Jun-jie-Huang/WhiteningBERT .

## introduction pre-trained language models (plms) @xcite perform well on learning sentence semantics when fine-tuned with supervised data @xcite . however, in practice, especially when a large amount of supervised data is unavailable, an approach that provides sentence embeddings in an unsupervised way is of great value in scenarios like sentence matching and retrieval. while there are attempts on unsupervised sentence embeddings @xcite , to the best of our knowledge, there is no comprehensive study on various plms with regard to multiple factors. meanwhile, we aim to provide an easy-touse toolkit that can be used to produce sentence embeddings upon various plms. in this paper, we investigate plms-based unsupervised sentence embeddings from three aspects. first, a standard way of obtaining sentence embedding is to pick the vector of [cls] token. we explore whether using the hidden vectors of other tokens is beneficial. second, some works suggest producing sentence embedding from the last layer or the combination of the last two layers @xcite . we seek to figure out whether there exists a better way of layer combination. third, recent attempts transform sentence embeddings to a different distribution with sophisticated networks @xcite to address the problem of non-smooth anisotropic distribution. instead, we aim to explore whether a simple linear transformation is sufficient. to answer these questions, we conduct thorough experiments upon 4 different plms and evaluate on 7 datasets regarding semantic textual similarity. we find that, first, to average the token representations consistently yields better sentence representations than using the representation of the [cls] token. second, combining the embeddings of the bottom layer and the top layer performs than using top two layers. third, normalizing sentence embeddings with whitening, an easy linear matrix transformation algorithm with less than 10 lines of code ( §a.3), consistently brings improvements. ## transformer-based plms multi-layer transformer architecture @xcite has been widely used in pre-trained language models (e.g. @xcite to encode sentences. given an input sequence @xmath0, s 2 , . . . , s n }, a transformerbased plm produces a set of hidden representations h (0) , h (1) , . . . , h (l) , where in this paper, we use four transformer-based plms to derive sentence embeddings, i.e. bertbase @xcite , roberta-base @xcite , distilbert @xcite , and labse @xcite . they vary in the model architecture and pre-training objectives. specifically, bert-base, roberta-base, and labse follow an architecture of twelve layers of transformers but distilbert only contains six layers. additionally, labse is pre-trained with a unique translation ranking task which forces the sentence embeddings of a parallel sentence pair to be closer, while the other three plms do not include such a pre-training task for sentence embeddings. ## whiteningbert in this section, we introduce the three strategies to derive the sentence embedding s from plms. ## experiment we evaluate sentence embeddings on the task of unsupervised semantic textual similarity. we show experimental results and report the best way to derive unsupervised sentence embedding from plms. ## related works unsupervised sentence embeddings are mainly composed with pre-trained (contextual) word embeddings @xcite . recent attempts can be divided into two categories, according to whether the pre-trained embeddings are further trained or not. for the former, some works leverage unlabelled natural language inference datasets to train a sentence encoder without direct supervision @xcite @xcite . for the latter, some works propose weighted average word embeddings based on word features @xcite @xcite and kuo, 2020). however, these approaches need further training or additional features, which limits the direct applications of sentence embeddings in real-world scenarios. finally, we note that concurrent to this work, @xcite also explored whitening sentence embedding, released to arxiv one week before our paper. ## conclusion in this paper, we explore to find a simple and effective way to produce sentence embedding upon various plms. through exhaustive experiments, we make three empirical conclusions here. first, averaging all token representations consistently induces better sentence representations than using the [cls] token embedding. second, combining the embeddings of the bottom layer and the top layer outperforms that using the top two layers. third, normalizing sentence embeddings with a whitening algorithm consistently boosts the performance.