BERTRAM : Improved Word Embeddings Have Big Impact on Contextualized Model Performance (2020)

Abstract: Pretraining deep language models has led to large performance gains in NLP. Despite this success, Schick and Schütze (2020) recently showed that these models struggle to understand rare words. For static word embeddings, this problem has been addressed by separately learning representations for rare words. In this work, we transfer this idea to pretrained language models: We introduce BERTRAM, a powerful architecture based on BERT that is capable of inferring high-quality embeddings for rare words that are suitable as input representations for deep language models. This is achieved by enabling the surface form and contexts of a word to interact with each other in a deep architecture. Integrating BERTRAM into BERT leads to large performance increases due to improved representations of rare and medium frequency words on both a rare word probing task and three downstream tasks.

## introduction as word embedding algorithms (e.g. @xcite are known to struggle with rare words, several techniques for improving their representations have been proposed. these approaches exploit either the contexts in which rare words occur @xcite @xcite , their surfaceform @xcite @xcite , or both @xcite . however, all of this prior work is designed for and evaluated on uncontextualized word embeddings. contextualized representations obtained from pretrained deep language models (e.g. @xcite @xcite already handle rare words implicitly using methods such as byte-pair encoding @xcite , wordpiece embeddings @xcite and character-level cnns @xcite . nevertheless, schick and schütze (2020) recently showed that bert's @xcite performance on a rare word probing task can be significantly improved by explicitly learning representations of rare words using attentive mimicking (am) @xcite . however, am is limited in two important respects: these limitations apply not only to am, but to all previous work on obtaining representations for rare words by leveraging form and context. while using bag-of-words models is a reasonable choice for static embeddings, which are often themselves bagof-words (e.g. @xcite , it stands to reason that they are not the best choice to generate input representations for position-aware, deep language models. to overcome these limitations, we introduce bertram (bert for attentive mimicking), a novel architecture for learning rare word representations that combines a pretrained bert model with am. as shown in figure 1 , the learned rare word representations can then be used as an improved input representation for another bert model. by giving bertram access to both surface form and contexts starting at the lowest layer, a deep integration of both input signals becomes possible. assessing the effectiveness of methods like bertram in a contextualized setting is challenging: while most previous work on rare words was evaluated on datasets explicitly focusing on rare words (e.g @xcite @xcite , these datasets are tailored to uncontextualized embeddings and thus not suitable for evaluating our model. furthermore, rare words are not well represented in commonly used downstream task datasets. we therefore introduce rarification, a procedure to automatically convert evaluation datasets into ones for which rare words are guaranteed to be important. this is achieved by replacing task-relevant frequent words with rare synonyms obtained using semantic resources such as wordnet @xcite . we rarify three common text (or text pair) classification datasets: mnli @xcite ), ag's news @xcite and dbpedia @xcite . bertram outperforms previous work on four english datasets by a large margin: on the three rarified datasets and on wnlampro (schick and schütze, 2020). in summary, our contributions are as follows: ## related work surface-form information (e.g., morphemes, characters or character n-grams) is commonly used to improve word representations. for static word embeddings, this information can either be injected into a given embedding space @xcite , or a model can directly be given access to it during training @xcite @xcite . in the area of contextualized representations, many architectures employ subword segmentation methods (e.g. @xcite @xcite . others use riding a un ##ic ##y ##cle is hard bert a riding is hard bert bertram bertram unicycle figure 1: top: standard use of bert. bottom: our proposal; first bertram learns an embedding for "unicycle" that replaces the wordpiece sequence. bert is then run on this improved input representation. several datasets focus on rare words, e.g., stanford rare word @xcite , definitional nonce @xcite , and contextual rare word @xcite . however, unlike our rarified datasets, they are only suitable for evaluating uncontextualized word representations. rarification is related to adversarial example generation (e.g. @xcite , which manipulates the input to change a model's prediction. we use a similar mechanism to determine which words in a given sentence are most important and replace them with rare synonyms. ## dataset rarification the ideal dataset for measuring the quality of rare word representations would be one for which the accuracy of a model with no understanding of rare words is 0% whereas the accuracy of a model that perfectly understands rare words is 100%. unfortunately, existing datasets do not satisfy this desidera-tum, not least because rare words -by their nature -occur rarely. this does not mean that rare words are not important: as we shift our focus in nlp from words and sentences as the main unit of processing to larger units like paragraphs and documents, rare words will occur in a high proportion of such larger "evaluation units". rare words are also clearly a hallmark of human language competence, which should be the ultimate goal of nlp. our work is part of a trend that sees a need for evaluation tasks in nlp that are more ambitious than what we have now. 4 to create more challenging datasets, we use rarification, a procedure that automatically transforms existing text classification datasets in such a way that rare words become important. we require a pretrained language model m as a baseline, an arbitrary text classification dataset d containing labeled instances (x, y) and a substitution dictionary s, mapping each word w to a set of rare synonyms s(w). given these ingredients, our procedure consists of three steps: (i) splitting the dataset into a train set and a set of test candidates, (ii) training the baseline model on the train set and (iii) modifying a subset of the test candidates to generate the final test set. ## conclusion we have introduced bertram, a novel architecture for inducing high-quality representations for rare words in bert's and roberta's embedding spaces. this is achieved by employing a powerful pretrained language model and deeply integrating surface-form and context information. by replacing important words with rare synonyms, we created downstream task datasets that are more challenging and support the evaluation of nlp models on the task of understanding rare words, a capability that human speakers have. on all of these datasets, bertram improves over standard bert and roberta, demonstrating the usefulness of our method. our analysis showed that bertram is beneficial not only for rare words (our main target in this paper), but also for frequent words. in future work, we want to investigate bertram's potential benefits for such frequent words. furthermore, it would be interesting to explore more complex ways of incorporating surface-form information -e.g., by using a character-level cnn similar to the one of @xcite -to balance out the potency of bertram's form and context parts.