Contextual Embeddings: When Are They Worth It? (2020)

Abstract: We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline—random word embeddings—focusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of data for which contextual embeddings give particularly large gains: language containing complex structure, ambiguous word usage, and words unseen in training.

## introduction in recent years, rich contextual embeddings such as elmo @xcite and bert @xcite have enabled rapid progress on benchmarks like glue @xcite and have seen widespread industrial use @xcite . however, these methods require significant computational resources (memory, time) during pretraining, and during downstream task training and inference. thus, an important research problem is to understand when these contextual embeddings add significant value vs. when it is possible to use more efficient representations without significant degradation in performance. as a first step, we empirically compare the performance of contextual embeddings with classic embeddings like word2vec @xcite and glove @xcite . to further understand what performance gains are attributable to improved embeddings vs. the powerful downstream models that leverage them, we also compare with a simple baseline-fully random embed- in our study on the impact of training set size, we find in experiments across a range of tasks that the performance of the non-contextual embeddings (glove, random) improves rapidly as we increase the amount of training data, often attaining within 5 to 10% accuracy of bert embeddings when the full training set is used. this suggests that for many tasks these embeddings could likely match bert given sufficient data, which is precisely what we observe in our experiments with industry-scale data. given the computational overhead of contextual embeddings, this exposes important trade-offs between the computational resources required by the embeddings, the expense of labeling training data, and the accuracy of the downstream model. to better understand when contextual embeddings give large boosts in performance, we identify three linguistic properties of nlp tasks which help explain when these embeddings will provide gains: intuitively, these properties distinguish between nlp tasks involving simple and formulaic text (e.g., assistant commands) vs. more unstructured and lexically diverse text (e.g., literary novels). we show on both sentiment analysis and ner tasks that contextual embeddings perform significantly better on more complex, ambiguous, and unseen language, according to proxies for these properties. thus, contextual embeddings are likely to give large gains in performance on tasks with a high prevalence of this type of language. ## background we discuss the different types of word embeddings we compare in our study: contextual pretrained embeddings, non-contextual pretrained embeddings, and random embeddings; we also discuss the relative efficiency of these embedding methods, both in terms of computation time and memory (sec. 2.1). ## experiments we provide an overview of our experimental protocols (section 3.1), the results from our study on the impact of training set size (section 3.2), and the results from our linguistic study (section 3.3). we show that the gap between contextual and noncontextual embeddings often shrinks as the amount of data increases, and is smaller on language that is simpler based on linguistic criteria we identify. ## related work the original work on elmo embeddings @xcite showed that the gap between contextual and non-contextual embeddings narrowed as the amount of training data increased. our work builds on these results by additionally comparing with random embeddings, and by studying the linguistic properties of tasks for which the contextual embeddings give large gains. our work is not the first to study the downstream performance of embeddings which do not require any pretraining. for example, in the context of neural machine translation (nmt) it is well-known that randomly-initialized embeddings can attain strong performance @xcite ; the work of @xcite empirically compares the performance of pretrained and randomlyinitialized embeddings across numerous languages and dataset sizes on nmt tasks, showing for example that the pretrained embeddings typically perform better on similar language pairs, and when the amount of training data is small (but not too small). furthermore, as mentioned in section 2, random embeddings were considered as a baseline by @xcite , to better understand the gains from using generic vs. domain-specific word embeddings for text classification tasks. in contrast, our goal for using random embeddings in our study was to help clarify when and why pretraining gives gains, and to expose an additional operating point in the trade-off space between computational cost, data-labeling cost, and downstream model accuracy. ## conclusion we compared the performance of contextual embeddings with non-contextual pretrained embeddings and with an even simpler baseline-random embeddings. we showed that these non-contextual embeddings perform surprisingly well relative to the contextual embeddings on tasks with plentiful labeled data and simple language. while much recent and impressive effort in academia and industry has focused on improving state-of-the-art performance through more sophisticated, and thus increasingly expensive, embedding methods, this work offers an alternative perspective focused on realizing the trade-offs involved when choosing or designing embedding methods. we hope this work inspires future research on better understanding the differences between embedding methods, and on designing simpler and more efficient models.