Phrase- BERT : Improved Phrase Embeddings from BERT with an Application to Corpus Exploration (2021)

Abstract: Phrase representations derived from BERT often do not exhibit complex phrasal compositionality, as the model relies instead on lexical similarity to determine semantic relatedness. In this paper, we propose a contrastive fine-tuning objective that enables BERT to produce more powerful phrase embeddings. Our approach (Phrase-BERT) relies on a dataset of diverse phrasal paraphrases, which is automatically generated using a paraphrase generation model, as well as a large-scale dataset of phrases in context mined from the Books3 corpus. Phrase-BERT outperforms baselines across a variety of phrase-level similarity tasks, while also demonstrating increased lexical diversity between nearest neighbors in the vector space. Finally, as a case study, we show that Phrase-BERT embeddings can be easily integrated with a simple autoencoder to build a phrase-based neural topic model that interprets topics as mixtures of words and phrases by performing a nearest neighbor search in the embedding space. Crowdsourced evaluations demonstrate that this phrase-based topic model produces more coherent and meaningful topics than baseline word and phrase-level topic models, further validating the utility of Phrase-BERT.

## introduction learning representations of phrases is important for many tasks, such as semantic parsing @xcite , machine translation @xcite , and question answering @xcite . while pretrained language models such as bert @xcite have significantly pushed forward the state of the art on a variety of nlp tasks, they still struggle to produce semantically meaningful embeddings for shorter linguistic units such as sentences and phrases. an out-of-the-box bert sentence embedding model often underperforms simple baselines such as averaging glove vectors in semantic textual similarity tasks @xcite ## phrase embeddings from bert we design two separate fine-tuning tasks on top of bert to improve its ability to produce meaningful phrase embeddings. since the pretrained bert model relies heavily on lexical overlap to determine phrase similarity, our first fine-tuning objective relies on an automatically generated dataset of lexically diverse phrasal paraphrases to encourage the model to move beyond string matching. the second objective encourages the model to encode contextual information into phrase embeddings by relying on a phrase-in-context dataset we extract of phrases in context from the huge-scale books3 corpus @xcite . in both cases, we rely on contrastive objectives similar to sentence-bert @xcite for fine-tuning (fig. 1 ). using bert to embed phrases: given an input phrase x of length n tokens, we compute a representation x by averaging the final-layer token-level vectors yielded by bert @xcite 2 after passing the tokens of x to the model as input: @xmath0. as all of bert's pretraining examples are 512 tokens long, the above method is reliable for short documents, but it struggles to model the semantics of words and phrases, as shown by @xcite and also by our evaluations in section 4.1. creating lexically diverse phrase-level paraphrases: our first fine-tuning objective encourages bert to capture semantic relatedness between phrases without overly relying on lexical similarity between those phrases. to accomplish this, we create a dataset by extracting 100k phrases from wikitext-103 @xcite using the shift-reduce parser from corenlp @xcite . 3 then, given a phrase p "complete control" from the sentence "the local authorities have complete control over the allocation of building materials", we create a positive example p + by passing p through the gpt2-based diverse paraphrasing model released by @xcite . this model was trained by fine-tuning gpt2-large @xcite on a filtered version of the paranmt-50m sentence-level paraphrase dataset @xcite , using an encoder-free seq2seq modeling approach as proposed by @xcite . we decode from this model using nucleus sampling with the nucleus probability mass of 0.8 @xcite , applying lexical constraints to avoid producing any non-stopword tokens that also occur in p. this yields phrases such as "full power of the system", which are quasiparaphrases of p with no lexical overlap. we create a negative example p -by first randomly sampling a non-stopword from p and then replacing it with a random token from the vocabulary. in the case of "complete control", we might sample "complete" and replace it with a randomly selected token "fluid". then, we feed the corrupted phrase into the paraphraser and decode just as we did to produce the positive example, which removes lexical overlap but preserves the distorted meaning. this produces phrases like "no change to the water level", which has no semantic relation to p. collecting phrases in context: the above dataset focuses exclusively on phrases out of context. in other words, a model trained to distinguish negative phrases from positive phrases does not observe any surrounding context in which these phrases are used. as these contexts also provide useful information about the meaning and usage of 3 we extract all np, vp, adjp, and advp phrases and then filter to select the most frequent 100k. more details on this process can be found in the appendix §a.1. positive context: ...... a wise decision to hire sarah because she's a [mask]. she helped on so many projects for various tasks...... negative context: ...... the nation is open to coast [mask] , exports large quantities of hemp, raises, rice and oil products ...... contrastive objective positive phrase: a worker with versatile skills anchor phrase: jack of all trades negative phrase: agreement on trade push close push apart figure 1: we propose phrase-bert to correct bert's embedding space for phrases by placing semantically similar phrases and contexts closer together (e.g., the anchor phrase "jack of all trade" and the positive paraphrase "a worker with versatile skills") and by removing lexical cues with a contrastive learning objective. we also store a single positive context c + of length 120 tokens in which p occurs, replacing the occurrence of p within c + with a mask token. ## experimental setup we evaluate our phrase embeddings on a diverse collection of phrase-level semantic relatedness tasks following previous works on evaluating phrase embeddings @xcite @xcite . due to a lack of benchmarks like senteval @xcite at the phrase level, we create filtered versions of some datasets by removing lexical overlap cues. ## results & discussion in this section, we highlight takeaways from our results on the phrase-level semantic relatedness benchmarks as well as measurements of lexical diversity. we also provide an ablation study that confirms the benefits of both fine-tuning objectives. ## conclusion we propose phrase-bert, which induces powerful phrase embeddings by fine-tuning bert with two contrastive objectives on datasets of lexically diverse phrase-level paraphrases and phrases-incontext. phrase-bert consistently outperforms strong baseline models on a suite of phrasal semantic relatedness tasks, even when lexical overlap cues are removed. these results suggest that phrase-bert looks beyond simple lexical overlap to capture complex phrase semantics. finally, we integrate phrase-bert into a neural topic model to enable phrase-based topic interpretation, and show that the resulting topics are more coherent and meaningful than competing methods.