Who’s on First?: Probing the Learning and Representation Capabilities of Language Models on Deterministic Closed Domains (2021)

Abstract: The capabilities of today’s natural language processing systems are typically evaluated using large datasets of curated questions and answers. While these are critical benchmarks of progress, they also suffer from weakness due to artificial distributions and incomplete knowledge. Artifacts arising from artificial distributions can overstate language model performance, while incomplete knowledge limits fine-grained analysis. In this work, we introduce a complementary benchmarking approach based on SimPlified Language Activity Traces (SPLAT). SPLATs are corpora of language encodings of activity in some closed domain (we study traces from chess and baseball games in this work). SPLAT datasets use naturally-arising distributions, allow the generation of question-answer pairs at scale, and afford complete knowledge in their closed domains. We show that language models of three different architectures can answer questions about world states using only verb-like encodings of activity. Our approach is extensible to new language models and additional question-answering tasks.

## introduction language models (lms) are pre-trained on running text and have been shown to have a remarkable ability to transform this raw observational data into models that capture syntax, semantics, and certain forms of world knowledge such as common sense @xcite . however, accurately measuring the models' representation and learning capabilities can be difficult because the tasks that they are evaluated on can be subjective, require expensive annotation, and suffer from dataset artifacts that allow models to achieve spurious high performance. in this paper, we propose novel, complementary benchmarks for evaluating the potential of different language model architectures. our approach involves pre-training and evaluating models on what we term simplified language activity traces (splat): corpora of language encodings of activity in a closed domain. splats have three characteristics that make them valuable for probing the capabilities of lms: (a) they are comprised of data from existing, organic distributions not designed by the experimenters, helping to prevent spurious artifacts, (b) they can be acquired at scale, enabling the construction of well-trained models, and (c) they admit automatic construction of questions with known, objectively correct answers, that probe the linguistic or cognitive potential of the models. we introduce two splat datasets based on language-like traces of chess and baseball games (see figure 1 ). the tokens in the datasets correspond to actions in the domain. for example, moving a knight from square g1 to square f3 is encoded as the splat token g1f3 (see appendix a). a sequence of splat tokens can represent entire games in each domain. we pre-train models for each domain on corpora of raw game traces, and then probe the models' capabilities by fine-tuning on synthetic questionanswering data sets. the mechanics of chess and baseball can be captured by simple symbolic programs, meaning that, in contrast to real natural language text, splats afford complete knowledgethat is, it is possible to unambiguously determine the relevant world (game) state at any point in the activity trace. we show how this enables us to create questions that probe key model capabilities, including the ability to construct and maintain world states, learn the rules that govern state transitions, synthesize across states (to determine who is winning), and generate multi-step paths of states to reach a given goal. further, due to the longstanding popularity of chess and baseball, traces for pretraining and fine-tuning can be obtained at scale. 1 many natural language processing (nlp) benchmarks are built from large data sets of humanannotated natural language questions. these data sets often contain artifacts @xcite , can be subjective, and are expensive to scale. prominent examples include glue @xcite , decanlp @xcite , squad 2.0 @xcite and coqa @xcite , among many others. a number of other benchmarks use natural or artificial language in synthetic environments with well-defined underlying mechanisms @xcite @xcite . tasks formulated on these synthetic datasets can probe complex capabilities like state-tracking, can be generated at scale, and have unambiguous answers-but the mechanisms and their distributional characteristics are designed by the experimenters. splats by contrast @xcite , decanlp @xcite , squad 2.0 @xcite and coqa @xcite , among others. synthetic language qa includes benchmarks like babi @xcite and other artificial language-based tasks @xcite @xcite . unsupervised lms include autoregressive and masked language models like gpt-2 @xcite and t5 @xcite , among others. splat benchmarks fulfill all of our desiderata, making them complementary to the human-annotated and synthetic datasets used for nlp tasks. results on these probing tasks show that language models perform surprisingly well, surpassing random baselines by wide margins in almost all cases. however, the models still fall short of the performance that could be achieved by short, human-authored symbolic programs for each domain, highlighting room for improvements and new architectures in language modeling. our contributions are twofold: ## related work multiple benchmark tasks exist to probe the representation and learning capabilities of language models. ## methods in this section we present the methodology and motivation used to design our probing tasks. one of our objectives is to provide an extensible framework for other researchers to evaluate new models and create new probes on splat. ## discussion and analysis to better understand two important aspects of our results, we conduct additional analyses of (i) a language models' ability to learn the "rules" of each game, and (ii) the extent to which memorization drives performance on probing tasks. ## conclusion we introduce the splat approach and novel chess and baseball datasets, and use them to probe language models' learning and representation capabilities. we perform these probes by pre-training language models on splat encodings of gameplay and then fine-tuning on question-answers pairs constructed to evaluate language model capabilities. probes are designed to evaluate a language model's ability to learn the "rules" of each game, construct and maintain world-states, synthesize across states and generate multi-step paths to each a given goal. we show that language models of different architectures possess the learning and representation capabilities to perform well on these tasks. these datasets are different from existing datasets in that (i) question-answer pairs can be generated at scale while maintaining the natural distribution properties of the gameplay data, (ii) accompanying symbolic programs can map verbs to deterministic changes in world-state, and (iii) complete knowledge is available throughout the question-answering process. we demonstrate a probing methodology that is extensible to different language models and additional probing questions. lastly, we note that splats can be constructed for other domains such as woz dialogues @xcite or quantitative behavior finance @xcite . rowan zellers, yonatan bisk, roy schwartz, and yejin choi. 2018. swag: a large-scale adversarial dataset for grounded commonsense inference. in emnlp.