A Primer in BERT ology: What We Know About How BERT Works (2020)

Abstract: Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.

## introduction since their introduction in 2017, transformers @xcite have taken nlp by storm, offering enhanced parallelization and better modeling of long-range dependencies. the best known transformer-based model is bert @xcite ; it obtained state-of-the-art results in numerous benchmarks and is still a must-have baseline. although it is clear that bert works remarkably well, it is less clear why, which limits further hypothesis-driven improvement of the architecture. unlike cnns, the transformers have little cognitive motivation, and the size of these models limits our ability to experiment with pre-training and perform ablation studies. this explains a large number of studies over the past year that attempted to understand the reasons behind bert's performance. in this paper, we provide an overview of what has been learned to date, highlighting the questions that are still unresolved. we first consider the linguistic aspects of it, namely, the current evidence regarding the types of linguistic and world knowledge learned by bert, as well as where and how this knowledge may be stored in the model. we then turn to the technical aspects of the model and provide an overview of the current proposals to improve bert's architecture, pre-training, and fine-tuning. we conclude by discussing the issue of overparameterization, the approaches to compressing bert, and the nascent area of pruning as a model analysis technique. the conventional workflow for bert consists of two stages: pre-training and fine-tuning. pretraining uses two self-supervised tasks: masked language modeling (mlm, prediction of randomly masked input tokens) and next sentence prediction (nsp, predicting if two input sentences are adjacent to each other). in fine-tuning for downstream applications, one or more fully connected layers are typically added on top of the final encoder layer. the input representations are computed as follows: each word in the input is first tokenized into wordpieces @xcite , and then three embedding layers (token, position, and segment) are combined to obtain a fixed-length vector. special token @xcite is used for classification predictions, and [sep] separates input segments. google 1 and huggingface @xcite ) provide many variants of bert, including the original ''base'' and ''large'' versions. they vary in the number of heads, layers, and hidden state size. a number of studies have looked at the knowledge encoded in bert weights. the popular approaches include fill-in-the-gap probes of mlm, analysis of self-attention weights, and probing classifiers with different bert representations as inputs. @xcite showed that bert representations are hierarchical rather than linear, that is, there is something akin to syntactic tree structure in addition to the word order information. @xcite and @xcite also showed that bert embeddings encode information about parts of speech, syntactic chunks, and roles. enough syntactic information seems to be captured in the token embeddings themselves to recover syntactic trees @xcite @xcite , although probing classifiers could not recover the labels of distant parent nodes in the syntactic tree @xcite . @xcite report evidence of hierarchical structure in three out of four probing tasks. ## training bert this section reviews the proposals to optimize the training and architecture of the original bert. ## directions for further research bertology has clearly come a long way, but it is fair to say we still have more questions than answers about how bert works. in this section, we list what we believe to be the most promising directions for further research. ## conclusion in a little over a year, bert has become a ubiquitous baseline in nlp experiments and inspired numerous studies analyzing the model and proposing various improvements. the stream of papers seems to be accelerating rather than slowing down, and we hope that this survey helps the community to focus on the biggest unresolved questions.