P rompt BERT : Improving BERT Sentence Embeddings with Prompts (2022)

Abstract: We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analysis the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings .Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.

## introduction in recent years, we have witnessed the success of pre-trained language models like bert @xcite and roberta @xcite in sentence embeddings @xcite . however, original bert still shows poor performance in sentence embeddings @xcite . the most commonly used example is that it underperforms the traditional word embedding methods like glove @xcite . previous research has linked anisotropy to explain the poor performance of original bert @xcite @xcite . anisotropy makes the token embeddings occupy a narrow cone, resulting in a high similarity between * work done during internship at microsoft. ## related work learning sentence embeddings as a fundamental nlp problem has been largely studied. currently, how to leverage the power of bert in sentence embeddings has become a new trend. many works @xcite achieved strong performance with bert in both supervised and unsupervised settings. among these works, contrastive learning based methods achieve state-ofthe-art results. @xcite proposed a novel contrastive training objective to directly use inner dropout as noise to construct positive pairs. @xcite discussed four methods to construct positive pairs. although bert achieved success in sentence embeddings, original bert shows unsatisfactory performance @xcite ). one explanation is the anisotropy in original bert, which causes sentence pairs to have high similarity, some works @xcite focused on reducing the anisotropy by post-processing sentence embeddings. ## rethinking the sentence embeddings of the original bert previous works @xcite explained the poor performance of the original bert is mainly due to the learned anisotropic token embeddings space, where the token embeddings occupy a narrow cone. however, we find that anisotropy is not a key factor to inducing poor semantic similarity by examining the relationship between the aniostropy and performance. we think the main reasons are the ineffective bert layers and static token embedding biases. observation 1: original bert layers fail to improve the performance. in this section, we analyze the influence of bert layers by comparing the two sentence embedding methods: averaging static token embeddings (input of the bert layers) and averaging last layer (output of the bert layers). we report the sentence embedding performance and its sentence level anisotropy. to measure the anisotropy, we follow the work of @xcite to measure the sentence level anisotropy in sentence embeddings. let s i be a sentence that appears in corpus {s 1 , ..., s n }. the anisotropy can be measured as follows: observation 2: embedding biases harms the sentence embeddings performance. @xcite found that token embeddings can be biased to token frequency. similar problems have been studied in @xcite . the anisotropy in bert static token embeddings is sensitive to token frequency. therefore, we investigate whether embedding bias yields unsatisfactory performance of sentence embeddings. we observe that the token embeddings is not only biased by token frequency, but also subwords in wordpiece @xcite and case sensitive. as shown in figure 1 , we visualize these biases in the token embeddings of bert-baseuncased, bert-base-cased and robertabase. the token embeddings of three pre-trained models are highly biased by the token frequency, subword and case. the token embeddings can be roughly divided into three regions according to the subword and case biases: 1) the lowercase beginof-word tokens, 2) the uppercase begin-of-word tokens and 3) the subword tokens. for uncased pretrained model bert-base-uncased, the token embeddings also can be roughly divided into two regions: 1) the begin-of-word tokens, 2) the subword tokens. for frequency bias, we can observe that high frequency tokens are clustered, while low frequency tokens are dispersed sparsely in all three models @xcite . the begin-of-word tokens are more vulnerable to frequency than subword tokens in bert. however, the subword tokens are more vulnerable in roberta. previous works @xcite ) often link token embeddings bias to the token embedding anisotropy and argue it is the main reason for the bias. however, we believe the anisotropy is unrelated to the bias. the bias means the distribution of embedding is disturbed by some irrelevant information like token frequency, which can be directly visualized according to the pca. for the anisotropy, it means the whole embedding occupies a narrow cone in the high dimensional vector space, which cannot be directly visualized. to prove the negative impact of biases, we show the influence of biases to the sentence embeddings with averaging static token embeddings as sentence embeddings (without bert layers). the results of eliminating embedding biases are quite impressive on three pre-trained models in manually removing embedding biases is a simple method to improve the performance of sentence embeddings. however, if the sentence is too short, this is not an adequate solution, which may result in the omission of some meaningful words. ## prompt based sentence embeddings inspired by @xcite , we propose a prompt based sentence method to obtain sentence embeddings. by reformulating the sentence embedding task as the mask language task, we can effectively use original bert layers by leveraging the large-scale knowledge. we also avoid the embedding biases by representing sentences from [mask] tokens. however, unlike the text classification and question-answering tasks, the output in sentence embeddings is not the label tokens predicted by mlm classification head, but the vector to represent the sentence. we discuss the implementation of prompt based sentence embeddings through addressing the following two questions: 1) how to represent sentences with the prompt, and 2) how to find a proper prompt for sentence embeddings. based on these, we propose a prompt based contrastive learning method to fine-tuning bert on sentence embeddings. ## experiments we conduct experiments on sts tasks with non fine-tuned and fine-tuned bert settings. for non fine-tuned bert settings, we exploit the performance of original bert in sentence embeddings, which corresponds to the previous findings of the poor performance of original bert. for fine-tuned bert settings, we report the unsupervised and supervised results by fine-tuning bert with downstream tasks. the results of transfer tasks are in appendix c. ## conclusion in this paper, we analyzed the poor performance of original bert for sentence embeddings, and find original bert is underestimated in sentence embeddings due to inappropriate sentence representation methods. these methods suffer from static token embedding bias and do not effectively use the original bert layer. to better leverage bert in sentence embeddings, we propose a prompt-based sentence embedding method, which helps original bert achieve impressive performance in sentence embeddings. to further improve our method in finetuning, we proposed a contrastive learning method based on template denoising. our extensive experiments demonstrate the efficiency of our method on sts tasks and transfer tasks.