BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023)

Abstract: Obtaining sentence representations from BERT-based models as feature extractors is invaluable as it takes much less time to pre-compute a one-time representation of the data and then use it for the downstream tasks, rather than fine-tune the whole BERT. Most previous works acquire a sentence’s representation by passing it to BERT and averaging its last layer. In this paper, we propose that the combination of certain layers of a BERT-based model rested on the data set and model can achieve substantially better results. We empirically show the effectiveness of our method for different BERT-based models on different tasks and data sets. Specifically, on seven standard semantic textual similarity data sets, we outperform the baseline BERT by improving the Spearman’s correlation by up to 25.75% and on average 16.32% without any further training. We also achieved state-of-the-art results on eight transfer data sets by reducing the relative error by up to 37.41% and on average 17.92%.

## introduction learning sentence vector representations is a crucial problem in natural language processing (nlp) and has been widely studied in the literature @xcite @xcite . given a sentence, the goal is to acquire a vector that semantically and/or syntactically represents it. bert @xcite has set new state-of-theart records on many nlp tasks @xcite @xcite . however, this is achieved by fine-tuning all of bert's layers. the disadvantage of fine-tuning is that it is computationally expensive as even bert-base has 110m parameters; hence, pre-computing a representation of the data and using it for the downstream task is much less computationally expensive @xcite . moreover, for sentence-pair tasks, bert uses a cross-encoder; nonetheless, this setup is inappropriate for certain pair regression tasks, such as finding the most similar sentence in a data set to a specific sentence due to the large number of possible combinations @xcite . considering the aforementioned drawbacks, researchers have tried to derive fixed-sized sentence embeddings from bert or proposed new berts with the exact same architecture but different ways of training @xcite . after training, the resultant bert is used in a feature-based manner by passing the sentence to it and obtaining its embedding vector in different ways, such as averaging the last layer of bert. in this paper, we present a simple, yet effective and novel method called bert-lc (bert layers combination). bert-lc combines certain layers of bert in order to obtain the representation of a sentence. as we will show, this model significantly outperforms its correspondent bert baseline with no need of any further training. our work was inspired by @xcite , who show that different layers of bert carry different features, such as surface, syntactic, and semantic. we argue that each data set with its unique distribution might need a different set of features for its sentences, which can only be fully exploited by combining different layers of bert in an unsupervised way. our contributions are as follows: (1) we propose a new method called bert-lc that is capable of acquiring superior results by combining certain layers of bert instead of just the last layer, in an unsupervised manner. we also include the embedding layer, which was to our knowledge ignored in previous works. (2) we additionally show that our method improves @xcite and @xcite , which were specifically designed for obtaining sentence representations (opposite to bert and roberta @xcite ). (3) we developed an algorithm that speeds up the process of finding the best layer combination (among 2 13 layer combinations in base cases) by a factor of 189 times. (4) we propose an innovative method that integrates the layer combination method with the cls pooling head, improving the performance metrics for certain models. (5) we achieve state-of-the-art performance on the transfer tasks using layer combination. we demonstrate the superiority of our approach through conducting extensive experiments on seven standard semantic textual similarity (sts) data sets and eight transfer tasks. on the sts data sets, our method is able to outperform its corresponding baseline by up to 25.75% and on average 16.32% for bert-large-uncased. we also achieve the stateof-the-art performances on transfer tasks, reducing the previous best model's relative error rate by an average of 17.92% and up to 37.41%. ## related work learning sentence embeddings is a well-studied realm in nlp. there are mainly two methods used for this purpose: methods that use unlabeled data or labeled data. although the latter use labeled data, the target data sets and tasks on which they are tested are different from the training data set and task. early work on sentence embedding utilized the distributional hypothesis by predicting surrounding sentences of a sentence @xcite @xcite . pagliardini et al. 2018 build on the idea of word2vec @xcite using ngram embeddings. some researchers simply use the average of bert's last layer embeddings as the sentence embedding @xcite . recently, contrastive learning has proven to be very powerful in many domains @xcite @xcite ; thus, some recent methods exploit contrastive learning @xcite @xcite and utilize the same sentence by looking at it from multiple angles. for instance, the popular method simcse leverages different outputs of the same sentence from bert's standard dropout. there are some previous works @xcite @xcite that investigate the impact of different bert layers. however, they either investigate each layer individually or are restricted to considering a set of consecutive layers strictly starting from layer 1. we apply our method to different variations of bert, roberta, sbert, and simcse. to the best of our knowledge, no similar work has been done that whether combines different and arbitrary layers of these models or further integrates layer combination and the cls pooling head. ## approach given an input sentence @xmath0, s 2 , . . . , s n ), the goal of a sentence embedding model is to output a vector e @xmath1. in order to obtain a sentence embedding, we first pass the sentence to a bert-like model, which outputs the tensor @xmath2, in which d is the dimension of the token vectors in each layer, and l ′ = l+1 is the number of layers (including layer 0) in bert. we then apply to this tensor a pooling function p, which can be max or mean, but we choose mean as it yielded better results in our experiments. the pooling is done across all the tokens in all the desired layers set, d. for example, for mean pooling, p is defined as: as iterating through all possible ds is very timeconsuming, we propose an algorithm that speeds up the process of finding the best layer combination (2 13 layer combinations in base cases) by a factor of 189 times, which can be found in appendix a. we further propose an extension to our method, and that is exploiting the mlp head and layer combination simultaneously. the idea is to pass p(h, d) in eq. 1 to the mlp head and use the output of the mlp head as the new p. we spot that this method works better than merely using the layer combination on simcse. we conjecture that this is because simcse's mlp head was trained for learn-ing sentence embeddings as opposed to the other methods, such as bert, sbert, and roberta; hence, it carries important information to be utilized. consequently, we use the two-step pipeline of layer combination and mlp for simcse models, and we propose that the two-step pipeline is utilized for any other bert-based sentence embedding model whose mlp head has been trained for sentence embedding. ## experiments and results we carry out our experiments on two different tasks: transfer and sts tasks. in this section, we discuss the tranasfer tasks, while the sts tasks are discussed in appendix b. ## ablation studies while more ablation studies, such as the effect of the number of layers in layer combination and the effect of excluding a particular layer from layer combination, can be found in appendix e, in this section, we discuss the performance of layer combination in limited data settings. for this, we reduce the size of training data for mr, subj, and trec, while keeping the test data untouched. fig. 1 shows the results for roberta-b (rb), bert-b cased (bbc) and sroberta-b (srb) on these data sets. we report the average results of 10 different seeds. as we can see, for all the models on all the data sets, the more reduction in size is done, the more the difference is between the lc version and the last layer average. for instance, for rb on subj, on the original training data, the accuracies for rb-lc and rb are 94.77% and 93.73%, respectively, while when reducing the size to 1/256th, the respective accuracies are 78.99% and 50.83%. this suggests that it is even more beneficial to use lc when one has small training data. ## conclusion and future work in this paper, we proposed a new method called bert layer combination, a simple, yet effective framework, which when applied to various bert-based models, significantly improves them for the downstream tasks of sts and transfer learning. further, it achieves the state-of-the-art performances on eight transfer tasks. our method combines certain layers of bert-based models in an unsupervised manner, which shows that different layers of bert hold important information which was previously ignored. we demonstrated the effectiveness of our approach by conducting comprehensive experiments on various bert-based models and on a host of different tasks and data sets. as future work, we would like to apply the technique of layer combination to other nlp tasks (e.g., punctuation insertion (hosseini and sameti, 2017) and question answering @xcite ) and also utilize it in other domains where deep learning models are used (e.g., biosignal analysis @xcite and image captioning @xcite ).