Which * BERT ? A Survey Organizing Contextualized Encoders (2020)

Abstract: Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.

## introduction a couple years ago, @xcite won the naacl best paper award for creating strong performing, task-agnostic sentence representations due to large scale unsupervised pretraining. days later, its high level of performance was surpassed by @xcite which boasted representations beyond a single sentence and finetuning flexibility. this instability and competition between models has been a recurring theme for researchers and practitioners who have watched the rapidly narrowing gap between text representations and language understanding benchmarks. however, it has not discouraged research. given the recent flurry of models, we often ask: "what, besides state-of-the-art, does this newest paper contribute? which encoder should we use?" the goals of this survey are to outline the areas of progress, relate contributions in text encoders to ideas from other fields, describe how each area is evaluated, and present considerations for practitioners and researchers when choosing an encoder. this survey does not intend to compare specific model metrics, as tables from other works provide comprehensive insight. for example, we organize this paper as follows. §2 provides brief background on encoding, training, and evaluating text representations. §3 identifies and analyzes two classes of pretraining objectives. in §4, we explore faster and smaller models and architectures in both training and inference. §5 notes the impact of both quality and quantity of pretraining data. §6 briefly discusses efforts on probing encoders and representations with respect to linguistic knowledge. §7 describes the efforts into training and evaluating multilingual representations. within each area, we conclude with high-level observations and discuss the evaluations that are used and their shortcomings. we conclude in §8 by making recommendations to researchers: publicizing negative results in this area is especially important owing to the sheer cost of experimentation and to ensure evaluation reproducibility. in addition, probing studies need to focus not only on the models and tasks, but also on the pretraining data. we pose questions for users of contextualized encoders, like whether the compute requirement of a model is worth the benefits. we hope our survey serves as a guide for both nlp researchers and practitioners, orienting them to the current state of the field of contextualized encoders and differences between models. ## background encoders pretrained text encoders take as input a sequence of tokenized foot_0 text, which is encoded by a multi-layered neural model. the representation of each (sub)token, x t , is either the set of hidden weights, {h (l) t } for each layer l, or its weight on just the top layer, h (-1) t . unlike fixed-sized word, sentence, or paragraph representations, the produced contextualized representations of the text depends on the length of the input text. most encoders use the transformer architecture @xcite . ## area i: pretraining tasks to utilize data at scale, pretraining tasks are typically self-supervised. we categorize the contributions into two types: token prediction (over a large vocabulary space) and nontoken prediction (over a handful of labels). in this section, we discuss several empirical observations. while token prediction is clearly important, less clear is which variation of the token prediction task is the best (or whether it even matters). nontoken prediction tasks appear to offer orthogonal contributions that marginally improve the language representations. we emphasize that in this section, we seek to outline the primary efforts in pretraining objectives and not to provide a comparison on a set of benchmarks. foot_2 ## area ii: efficiency as models perform better but cost more to train, some have called for research into efficient models to improve deployability, accessibility, and reproducibility @xcite @xcite . encoders tend to scale effectively @xcite @xcite , so efficient models will also result in improvements over inefficient ones of the same size. in this section, we give an overview of several efforts aimed to decrease the computation budget (time and memory usage) during training and inference of text encoders. while these two axes are correlated, reductions in one axis do not always lead to reductions in the other. ## area iii: (pretraining) data unsurprisingly for our field, increasing the size of training data for an encoder contributes to increases in language understanding capabilities @xcite @xcite . at current data scales, some models converge before consuming the entire corpus. in this section, we identify a weakness when given less data, advocate for better data cleaning, and raise technical and ethical issues with using web-scraped data. ## area iv: interpretability while it is clear that the performance of text encoders surpass human baselines, it is less clear what knowledge is stored in these models; how do they make decisions? in their survey, @xcite find answers to the first question and also raise the second. inspired by prior work @xcite @xcite , we organize here the major probing methods that are applicable to all encoders in hopes that future work will use comparable techniques. ## conclusions in this survey we categorize research in contextualized encoders and discuss some issues regarding its conclusions. we cover background on contextualized encoders, pretraining objectives, efficiency, data, approaches in model interpretability, and research in multilingual systems. as there is now a large selection of models to choose from, we discuss tradeoffs that emerge between models. we hope this work provides some assistance to both those entering the nlp community and those already using contextualized encoders in looking beyond sota (and twitter) to make more educated choices.