Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction (2020)

Abstract: Contemporary autoregressive language models (LMs) trained purely on corpus data have been shown to capture numerous features of human incremental processing. However, past work has also suggested dissociations between corpus probabilities and human next-word predictions. Here we evaluate several state-of-the-art language models for their match to human next-word predictions and to reading time behavior from eye movements. We then propose a novel method for distilling the linguistic information implicit in human linguistic predictions into pre-trained LMs: Cloze Distillation. We apply this method to a baseline neural LM and show potential improvement in reading time prediction and generalization to held-out human cloze data.

## introduction modern language models (lms) demonstrate outstanding general-purpose command over language. the majority of these models acquire language by maximizing the in-context probability of each word in their training corpus (figure 1 ), typically with a self-supervised objective. this simple corpus probability matching has resulted in models that learn impressive powers of both psychometric prediction @xcite @xcite @xcite @xcite and language more generally @xcite . in humans, prediction may underlie both learning @xcite and processing @xcite @xcite . human linguistic prediction can be understood as not only lexical but also as taking place both above and below the word level @xcite ; parallel, i.e., predictive commitments are maintained over several linguistic units at once @xcite ; and graded, i.e., commitment is licensed to varying degrees based on features of the linguistic unit being predicted. rather than placing bets @xcite on which single word will come next, humans make many diffuse bets at multiple linguistic levels (e.g., syntactic, orthographic, lexical, etc.). surprisal theory @xcite describes the utility of the approach taken by the human language processor, as lexical prediction is often an ill-constrained classification problem -for agents with very large vocabularies (lms, humans), context is often not sufficiently constraining for high accuracy multiple, thousand-way classification decisions, but is typically constraining enough to accurately infer next-word features (such as part of speech, and semantic category). a large body of evidence demonstrates that these graded next-word predictions are reflected in human processing times @xcite @xcite as well as neural responses @xcite . corpus data are (imperfect) samples from the linguistic environment of a native speaker, and psycholinguistic data indicate that accurate prediction is important to efficient language comprehension. under the principle of rational analysis @xcite , it is thus to be expected that artificial language models trained on corpus data would correlate with human linguistic predictions and thus have good psychometric predictive accuracy. nevertheless, past work @xcite has suggested dissociations between corpus probabilities and human next-word estimates. here, we further investigate this relationship using artificial language models and the most extensive corpus of sequential cloze completions that we are aware of: the provo corpus (provo henceforth; @xcite . first, we use provo to test the psychometric performance of three state-of-the-art transformerbased @xcite lms -xlnet @xcite , transformer-xl @xcite , and gpt-2 @xcite -alongside a smaller 2-layer lstm @xcite trained on wikitext-103 @xcite , and a 5-gram lm baseline @xcite . we find that, while the transformer models achieve the lowest perplexity on provo and the best fit to the cloze data, the lstm model provides the best account of reading times in terms of raw correlation. these findings show a dissociation between recapitulating corpus statistics and mimicking human language processing, operationalized here with reading times. that is, models that minimize perplexity on next-word prediction do not necessarily provide the best account of reading times. second, based on these findings, we propose cloze distillation: a novel method for distilling linguistic information implicit in human cloze completions into pre-trained lms. we apply this method to the lstm model and show substantial improvement in reading time prediction and word frequency estimation, in addition to generalization to held-out human cloze data. ## human cloze predictions the objective for most modern lms is to compute a probability distribution over the model's vocabulary v for the likely next-word @xmath0. similarly, as humans process language, they make constant and implicit linguistic predictions. one commonly used measure of these predictions in humans is the cloze task. in its original form @xcite , the task involved masking a word or words in a source text passage and asking participants to provide words for the masked elements that would make the passage "whole again", a task structure adopted by contemporary masked language models @xcite . in experimental psycholinguistics, however, the most common version of the cloze task has involved presenting the beginning, or prefix, of a passage and having participants either complete it or provide the word that they think comes next (figure 1 ), a task more closely matching that of autoregressive language models @xcite . in this paper, we focus on this latter type of cloze task, which elicits samples from comprehenders' subjective next-word probability distributions @xcite . for any given prefix, we can estimate the cloze distribution of a typical native speaker from pooled cloze responses across a large number of participants @xcite , similar to how the fundamental output of an autoregressive language model is a vector of next-word probabilities. ## testing language models on provo the findings of @xcite highlight cloze as a useful test-bed for lms. specifically, a lm that employs predictions similar to those that underlie human language processing is expected to be a good model of human cloze responses. therefore, we evaluate here a suite of lms on their ability to match human cloze distributions. additionally, we use the lms' ability to predict reading times as a second measure of fit to human expectations, extending past work using lms to predict reading times @xcite . ## cloze distillation here, we show how to leverage these findings to improve the ability of lms to match human expectations, providing more appealing neural language models for human language processing. to this end, we propose cloze distillation: a method for using human next-word predictions as learning targets together with corpus statistics within a knowledge distillation framework. ## conclusion our analyses provide further evidence of a misalignment between language model estimates and human expectations. the method we provide: cloze distillation, demonstrates that shifting training incentives away from corpus probability toward psycholinguistic task-based modeling can result in better cognitive models and better language models. still, given several of our models predict reading times beyond the cloze data collected in provo ( one is that the cloze task produces data that are a more faithful reflection of the expectations deployed in human reading and are thus able to guide the models toward a fundamentally more human-like set of expectations -despite being under-sampled. if this is true and human subjec-tive next-word estimates also provide signal about next-word probabilities across corpora (reflecting the implicit knowledge speakers have learned about the statistics of their language), this would explain why cloze distillation improves next-word prediction accuracy on a new corpus (provo). another possibility is that the models we survey are fundamentally better than the cloze data at capturing the human expectations deployed in reading. though this would not explain the boost in performance we see in reading time prediction with cloze distillation, because several of our models predict reading times better than the cloze data itself, this can not yet be ruled out. we leave the further exploration of this to future work as largerscale collection of human cloze data allows. that said, the fact that we were able to induce appreciable adaptions in model behavior with such little data highlights the richly orienting information available in even noisy human predictions. though it is unclear how language users learn to make such sophisticated predictions (we provided this information to our model with direct supervision), our model's ability to learn from such small scale data highlights the potential utility of such predictions in a language acquisition setting -it seems that human predictions are strong enough to significantly bolster the signal in raw linguistic input abetting extensive adaption from relatively little data. as of now, the current dataset's scale restricts cloze distillation to use as a fine-tuning method. furthermore, we use simple lstms to perform a detailed analysis of cloze distillation with dense sampling in α and thorough cross-validation. it is possible that deploying cloze distillation during pre-training in large models (e.g., transformers) could result in models better able to learn the word features humans demonstrate knowledge of in their cloze responses and we leave the exploration of this to future work as well. methods such as cloze distillation provide an avenue forward for psycholinguists interested in taking lms seriously as candidate models of human language processing and to natural language processing researchers interested in reverse engineering and deploying insights from human sentence processing. cloze distillation highlights these goals as potentially mutually-reinforcing.