Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings (2023)

Abstract: Prior studies diagnose the anisotropy problem in sentence representations from pre-trained language models, e.g., BERT, without fine-tuning. Our analysis reveals that the sentence embeddings from BERT suffer from a bias towards uninformative words, limiting the performance in semantic textual similarity (STS) tasks. To address this bias, we propose a simple and efficient unsupervised approach, Diagonal Attention Pooling (Ditto), which weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. Ditto can be easily applied to any pre-trained language model as a postprocessing operation. Compared to prior sentence embedding approaches, Ditto does not add parameters nor requires any learning. Empirical evaluations demonstrate that our proposed Ditto can alleviate the anisotropy problem and improve various pre-trained models on the STS benchmarks.

## introduction pre-trained language models (plm) such as bert @xcite , roberta @xcite , and electra @xcite have achieved great success in a wide variety of natural language processing tasks. however, @xcite finds that sentence embeddings from the original bert underperform traditional methods such as glove @xcite . typically, an input sentence is first embedded by the bert embedding layer, which consists of token embeddings, segment embeddings, and position embeddings. the output is then encoded by the transformer encoder and the hidden states at the last layer are averaged to obtain the sentence embeddings. prior studies identify the anisotropy problem as a critical factor that harms bert-based sentence embeddings, as sentence embeddings from the original bert yield a high similarity between any sentence pair due to the narrow cone of learned embeddings @xcite . prior approaches for improving sentence embeddings from plms fall into three categories. the first category of approaches does not require any learning (that is, learning-free). @xcite argues that the anisotropy problem may be mainly due to the static token embedding bias, such as token frequency and case sensitivity. to address these biases, they propose the static remove biases avg. method which removes top-frequency tokens, subword tokens, uppercase tokens, and punctuations, and uses the average of the remaining token embeddings as sentence representation. however, this approach does not use the contextualized representations of bert and may not be effective for short sentences as it may exclude informative words. the prompt-based method (last manual prompt) @xcite uses a template to generate sentence embeddings. an example template is this sentence: "[x]" means @xcite ., where [x] denotes the original sentence and the last hidden states in the [mask] position are taken as sentence embeddings. however, this method has several drawbacks. (1) it increases the input lengths, which raises the computation cost. (2) it relies on using the [mask] token to obtain the sentence representation, hence unsuitable for plms not using @xcite tokens (e.g., electra). (3) the performance heavily depends on the quality of manual prompts which relies on human expertise (alternatively, optiprompt @xcite requires additional unsupervised contrastive learning). the second category of approaches fixes the parameters of the plm and improves sentence embeddings through post-processing methods that require extra learning. bert-flow @xcite addresses the anisotropy problem by introducing a flow-based generative model that transforms the bert sentence embedding distribution into a smooth and isotropic gaussian distribution. bertwhitening @xcite uses a whitening operation to enhance the isotropy of sentence representations. both bert-flow and bert-whitening require natural language inference (nli)/semantic textual similarity (sts) datasets to train the flow network or estimate the mean values and covariance matrices as the whitening parameters. the third category updates parameters of the plm by fine-tuning or continually pre-training the plm using supervised or unsupervised learning, which is computationally intensive. for example, sentence-bert (sbert) @xcite fine-tunes bert using a siamese/triplet network on nli and sts datasets. simcse @xcite explores contrastive learning. unsupervised simcse uses the same sentences with different dropouts as positives and other sentences as negatives, and supervised simcse explores nli datasets and treats entailment pairs as positives and contradiction pairs as hard negatives. in this work, we first analyze bert sentence embeddings. (1) we use a parameter-free probing method to analyze bert and sentence-bert (sbert) @xcite and find that the compositionality of informative words is crucial for generating high-quality sentence embeddings as from sbert. (2) visualization of bert attention weights reveals that certain selfattention heads in bert are related to informative words, specifically self-attention from a word to itself (that is, the diagonal values of the attention matrix). based on these findings, we propose a simple and efficient approach, diagonal attention pooling (ditto), to improve sentence embeddings from plm without requiring any learning (that is, ditto is a learning-free method). we find that ditto improves various plms and strong sentence embedding methods on sts benchmarks. ## analyze bert sentence embeddings observation 1: the compositionality of informative words is crucial for high-quality sentence embeddings. perturbed masking @xcite is a parameter-free probing technique for analyzing plms (e.g., bert). given a sentence @xmath0, x 2 , . . . , x n ], perturbed masking applies a two-stage perturbation process on each pair of tokens (x i , x j ) to measure the impact that a token x j has on predicting the other token x i . details of perturbed masking can be found in ap- observation 2: certain self-attention heads of bert correspond to word importance. although sbert has a higher correlation with tf-idf than bert as verified in observation 1, bert still shows a moderate correlation. thus, we hypothesize that the semantic information of informative words is already encoded in bert but has yet to be fully exploited. prior research @xcite analyzes the attention mechanisms of bert by treating each attention head as a simple, no-training-required classifier that, given a word as input, outputs the other word that it most attends to. certain attention heads are found to correspond well to linguistic notions of syntax and coreference. for instance, heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions are found to have remarkably high accuracy. we believe that the attention information in bert needs to be further exploited. we denote a particular attention head @xmath1, where for a bert-base-sized model, layer ranges from 1 to 12 and head number ranges from 1 to 12. we visualize the attention weights of each head in each layer of bert and focus on informative words. we then discover that self-attention from a word to itself (that is, the diagonal value of the attention matrix, named diagonal attention) of certain heads may be related to the importance of the word. as shown in figure 2 , the informative words "social media transitions", "hill", and "little" have high diagonal values of the attention matrix of head 1-10. section 4 will demonstrate that diagonal attentions indeed have a strong correlation with tf-idf weights. ## diagonal attention pooling inspired by the two observations in section 2, we propose a novel learning-free method called diagonal attention pooling (ditto) to improve sentence embeddings for plms, illustrated in figure 3 . taking bert as an example, the input to the transformer encoder is denoted as h 0 = [h 0 1 , . . . , h 0 n ], and the hidden states at each transformer encoder layer are denoted as h @xmath2, . . . , h l n ], @xmath3, . . . , l}. typically, the hidden states of the last layer of the plm are averaged to obtain the fixedsize sentence embeddings, as 1 n n @xmath4.). alternatively, we can also average the static word embeddings 1 n n @xmath5.), or average the hidden states from the first and last layers 1 2n n @xmath6. ditto weights the hidden states with diagonal attention of a certain head. for example, to obtain the sentence embeddings from the first-last hidden states of bert using ditto, we first obtain the diagonal values [a 11 , . . . , a n n ] of the attention matrix a for head l-h of bert, where l and h are treated as hyperparameters and are optimized on a development set based on the sts performance. then, we compute as the sentence embeddings foot_1 . note that the impact matrix (section 2) correlates well with tf-idf (as shown in ## conclusions we propose a simple and learning-free diagonal attention pooling (ditto) approach to address the bias towards uninformative words in bert sentence embeddings. ditto weights words with modelbased importance estimations and can be easily applied to various plms. experiments show that ditto alleviates the anisotropy problem and improves strong sentence embedding baselines.