o LM pics-On What Language Model Pre-training Captures (2020)

Abstract: Recent success of pre-trained language models (LMs) has spurred widespread interest in the language capabilities that they possess. However, efforts to understand whether LM representations are useful for symbolic reasoning tasks have been limited and scattered. In this work, we propose eight reasoning tasks, which conceptually require operations such as comparison, conjunction, and composition. A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data. To address this, we propose an evaluation protocol that includes both zero-shot evaluation (no fine-tuning), as well as comparing the learning curve of a fine-tuned LM to the learning curve of multiple controls, which paints a rich picture of the LM capabilities. Our main findings are that: (a) different LMs exhibit qualitatively different reasoning abilities, e.g., RoBERTa succeeds in reasoning tasks where BERT fails completely; (b) LMs do not reason in an abstract manner and are context-dependent, e.g., while RoBERTa can compare ages, it can do so only when the ages are in the typical range of human ages; (c) On half of our reasoning tasks all models fail completely. Our findings and infrastructure can help future work on designing new datasets, models, and objective functions for pre-training.

## introduction large pre-trained language models (lms) have revolutionized the field of natural language processing in the last few years @xcite @xcite @xcite . this has insti-gated research exploring what is captured by the contextualized representations that these lms compute, revealing that they encode substantial amounts of syntax and semantics @xcite @xcite @xcite . despite these efforts, it remains unclear what symbolic reasoning capabilities are difficult to learn from an lm objective only. in this paper, we propose a diverse set of probing tasks for types of symbolic reasoning that are potentially difficult to capture using a lm objective (see however, how does one verify whether pretrained representations hold information that is useful for a particular task? past work mostly resorted to fixing the representations and finetuning a simple, often linear, randomly initialized probe, to determine whether the representations hold relevant information @xcite @xcite @xcite @xcite . however, it is difficult to determine whether success is due to the pre-trained representations or due to fine-tuning itself @xcite . to handle this challenge, we include multiple controls that improve our understanding of the results. our ''purest'' setup is zero-shot: we cast tasks in the masked lm format, and use a pre-trained lm without any fine-tuning. for example, given the statement ''a cat is @xcite than a mouse'', an lm can decide if the probability of ''larger'' is higher than ''smaller'' for the a masked word (figure 1 ). if a model succeeds without pretraining over many pairs of objects, then its representations are useful for this task. however, if it fails, it could be due to a mismatch between the language it was pre-trained on and the language of the probing task (which might be automatically generated, containing grammatical errors). thus, we also compute the learning curve (figure 1 ), by fine-tuning with increasing amounts of data on the already pre-trained masked language modeling (mlm) output ''head'', a 1-hidden layer multilayer perceptron (mlp) on top of the model's contextualized representations. a model that adapts from fewer examples arguably has better representations for it. moreover, to diagnose whether model performance is related to pre-training or fine-tuning, we add controls to every experiment (figures 1 , 2 ). first, we add a control that makes minimal use of language tokens, that is, ''cat @xcite mouse'' (no lang. in figure 1 ). if a model succeeds given minimal use of language, the performance can be mostly attributed to fine-tuning rather than to the pre-trained language representations. similar logic is used to compare against baselines that are not pre-trained (except for non-contextualized word embeddings). overall, our setup provides a rich picture of whether lm representations help in solving a wide range of tasks. we introduce eight tasks that test different types of reasoning, as shown in our contributions are summarized as follows: ## models we now turn to the architectures and loss functions used throughout the different probing tasks. ## controlled experiments we now describe the experimental design and controls used to interpret the results. we use the age-compare task as a running example, where models need to compare the numeric value of ages. ## the olmpic games we now move to describe the research questions and various probes used to answer these questions. for each task we describe how it was constructed, show results via a table as described in the controls section, and present an analysis. our probes are mostly targeted towards symbolic reasoning skills ( ## medals we summarize the results of the olmpic games in analysis of correlation with pre-training data a possible hypothesis for why a particular model is successful in a particular task might be that the language of a probe is more common in the corpus it was pre-trained on. to check that, we compute the unigram distribution over the training corpus of both bert and roberta. we then compute the average log probability of the development set under these two unigram distributions for each task (taking into account only content words). finally, we compute the correlation between which model performs better on a probe (roberta-l vs. bert-wwm) and which training corpus induces higher average log probability on that probe. we find that the spearman correlation is 0.22, hinting that the unigram distributions do not fully explain the difference in performance. ## discussion we presented eight different tasks for evaluating the reasoning abilities of models, alongside an evaluation protocol for disentangling pre-training from fine-tuning. we found that even models that have identical structure and objective functions differ not only quantitatively but also qualitatively. specifically, roberta-l has shown reasoning abilities that are absent from other models. thus, with appropriate data and optimization, models can acquire from an lm objective skills that might be surprising intuitively. however, when current lms succeed in a reasoning task, they do not do so through abstraction and composition as humans perceive it. the abilities are context-dependent, if ages are compared-then the numbers should be typical ages. discrepancies from the training distribution lead to large drops in performance. last, the performance of lm in many reasoning tasks is poor. our work sheds light on some of the blind spots of current lms. we will release our code and data to help researchers evaluate the reasoning abilities of models, aid the design of new probes, and guide future work on pre-training, objective functions and model design for endowing models with capabilities they are currently lacking.