What Context Features Can Transformer Language Models Use? (2021)

Abstract: Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of experiments that measure usable information by selectively ablating lexical and structural information in transformer language models trained on English Wikipedia. In both mid- and long-range contexts, we find that several extremely destructive context manipulations—including shuffling word order within sentences and deleting all words other than nouns—remove less than 15% of the usable information. Our results suggest that long contexts, but not their detailed syntactic and propositional content, are important for the low perplexity of current transformer language models.

## introduction recent years have seen a significant improvement in the predictive accuracy of neural language models (lms), owing to a combination of improvements in model architecture (especially transformers; @xcite and training infrastructure @xcite . the most striking change, relative to both recurrent neural lms @xcite and count-based models @xcite , is the length of the context that these models can effectively condition on. while count-based lms in production speech recognition and machine translation systems typically used 10-20 tokens at a maximum (e.g., @xcite , and recurrent lms have an effective context size of 200 @xcite , the predictive accuracy of transformer lms appears to improve when conditioning on as many as a thousand previous tokens @xcite . a significant amount of recent work has focused on making use of even longer contexts computationally feasible @xcite @xcite @xcite . but despite empirical evidence that long contexts are helpful, little is understood about why. if the future of language modeling will include a focus on contexts of increasing size, it is important to first understand what contextual information contributes to accurate prediction in current models. this paper offers an answer to that question via the v-information framework of @xcite . vinformation, discussed more in section 2, provides a formal framework for reasoning about how much usable information a computationally constrained predictor (like a neural lm) can extract from an input. our experiments measure the amount of usable information that is added when increasing lm context size, then attempt to pinpoint the source of this information by ablating features of the added context (via controlled shuffling and word deletion) and measuring the resulting loss of model predictive power. while this framework is general, we focus on transformer lms. our work is closely related to an earlier study by @xcite , which measured changes in a pre-trained lstm lm when context words were permuted and deleted at evaluation time. but neural language models are known to be highly sensitive to distributional shifts-and in particular might be unable to use information from long-range context but still be adversely affected when the structure of that context changes at evaluation time. directly measuring usable information makes it possible to clearly distinguish accuracy decreases that result from loss of information and decreases that result from out-of-distribution inputs. our experiments reveal a number of surprising facts about the use of long-and mid-range context in transformers. while increasing context length from 256 to 768 tokens is beneficial (decreasing perplexity by roughly 4%), many destructive transformations of this context (including transformations that cause large changes in the paradigm of @xcite remove essentially no usable information. our results suggest that for current models, the primary carriers of information in long-range context are content words and local cooccurrence statistics: deleting function words and shuffling within local windows both have very little effect on models' predictive power. context matters, but not all features of context matter equally; as discussed in section 5, these results motivate future language modeling research focused on alternative context representations rather than simply more tokens. ## approach a language model (lm) places a probability distribution p(x) over discrete token sequences x. most learned lms do so by decomposing p(x) according to the chain rule and modeling the conditional distribution over a single target token given a (fixedor variable-length) context of previous tokens: in transformer language models, this conditional distribution is modeled via a sequence of alternating neural feed-forward layers and self-attention layers; see @xcite for more details. while input sequences x can in principle be made arbitrarily long, there are both theoretical and practical limits to transformers' ability to make effective use of it @xcite . here, we wish to understand when (and why) increasing the size of the context improves model predictions. usable information consider a hypothetical lm context consisting of the tokens the user's password is. . . . this context suggests that subsequent tokens will be a password: (hopefully!) a high-entropy sequence. now suppose this context is extended to include earlier tokens, becoming the user's hashed password is ave$@to9!. the user's password is. . . . information-theoretically, this context is extremely informative: only a small number of passwords will hash to the given string, and a predictor capable of testing all passwords would be able to identify the candidates and significantly reduce its uncertainty about future tokens. but in practice, this extra context is useless: no known efficient predictor can learn anything about the password from its hash code, and the extra context has not made the language modeling problem any easier. this is an extreme case, but a similar intuition applies to more conventional questions about language models. a newspaper article whose first sentence begins a dog bit a man is likely to end very differently from one that begins a man bit a dog. can lms reason effectively about this distinction, or is it (like a hashed password) computationally inaccessible to current models? a framework for answering questions of this kind was introduced by @xcite : definition 1. the usable predictive information (formally, predictive v-information) from a random variable x to a random variable y as: intuitively, this definition measures how much extra information about y can be extracted from x by any predictor in v. in language modeling, we will take y to be the target word, x its context, and v a class of parametric models. while this definition generalizes shannon mutual information @xcite and has deep connections to other information-theoretic quantities (see @xcite for details) it ultimately corresponds to a simple and common-sense evaluation: if we want to know how much the extra context x helps a language model, we should train a model p 1 without access to x, train a model p 2 with access to x, and compare the accuracy of their predictions. measuring what is used but the original question raised by the introduction was not just how much information is contributed by context. it is already well-established that conditioning on long contexts is helpful, with existing experiments on long-range transformers effectively implementing the measurement in eq. ( 2 ). instead, we want to know what information in this context is actually used by models. as a prototypical example, let us hypothesize that more than five tokens away from the target, models are only able to extract usable information from nouns. (in our experiments in section 3, this "long-range context" will be considerably longer than 5 words.) for example, given the sentence: pierre vinken, 61 years old, will join the board as a nonexecutive director nov. 29. the experiments in this paper generalize this experimental framework to other context ablations and hypotheses. let f be an ablation and k an integer offset, and denote an ablated context: then, we can measure the effect of each ablation f on usable information via the following quantity: definition 2. the ablated information due to an ablation f at an offset k is: intuitively, a(f, k) measures how much of the usable information added by an extra k tokens (the denominator) is removed by applying the ablation f to those k tokens (the numerator). if it is close to 0, almost no information is removed; if it is close to 1, almost all information is removed. 10 )). a context ablation nouns (which deletes all non-noun words) is applied to the first ℓ tokens of the context, and likelihood is computed on the last n -m (unablated) context tokens. evaluation in practice eq. ( 9 ) provides a general framework for answering our core question in this paper: for a diverse set of context ablations and offsets, we will measure how much information is lost when a given ablation is applied at a given offset. a few modifications are required to turn this equation into a practical evaluation scheme: held-out evaluation: eq. ( 7 ) involves an expectation over the sequence distribution p(x). in practice, lms must be trained on finite corpora, creating a risk of overfitting @xcite . to address this issue, we approximate the infimum in eq. ( 7 ) by fitting θ 1 on a training set, and computing ablated information on a held-out validation set. all reported results are an average of held-out likelihoods from two random initializations. batching: given a fixed (training or test) dataset of strings x and a maximum context size of m, eq. ( 7 ) should be estimated empirically as - model, data and training details for all experiments, our lm uses the gpt-2 model architecture @xcite in the implementation of @xcite with default hyperparameters. all models are trained from scratch on the wikitext-103 dataset @xcite , an english language modeling benchmark. aside from ablations, no preprocessing is applied. a special separator token is inserted between ablated and unablated context. the training set contains 103,221,021 words, while the evaluation set contains 217,646 words. a note on evaluation as in past work on evaluating language models @xcite , our evaluation of relative predictive information ultimately bottoms out in a conditional entropy (logperplexity). recent work has shown that other metrics, such as diversity of outputs, are important for evaluating the quality of lms as models for language generation @xcite . generation also depends on a number of other factors, such as choice of decoding procedure @xcite . here, we focus on lms as predictive models, measuring their ability to place an accurate distribution over future words and sentences, rather than their ability to generate useful or coherent text (see appendix c). we want to emphasize that these results below apply to language models specifically, and not transformers applied to nlp tasks in general-the same analysis might give very different conclusions if applied to, e.g., question answering or summarization. ## experiments in this section, we attempt to determine what information in transformer lm contexts is usable by measuring ablated information (eq. ( 9 )). sections 3.1 and 3.2 describe our main results, with section 3.1 focused on ordering and section 3.2 focused on lexical information. section 3.3 compares these results to ablations applied at evaluation time. section 3.4 explores whether contexts can be further manipulated to improve model predictions. ## related work context in count-based and discriminative lms the earliest learned lms were count-based (e.g., @xcite subsequent models improved on these by allowing the use of skip-grams, caches, and featurebased models @xcite . some of these in principle allowed the use of unlimited-length contexts, but only by imposing strong restrictions on the ways in which context features could interact. context in rnn lms recurrent neural network language models @xcite provide a more expressive mechanism for the use of long-range context: models write to a recurrent "state vector" which can be carried arbitrarily far into the future. computational issues limit the effective context size such models can be practically trained on, but this size is still significantly greater the models mentioned above: as previously noted, @xcite revealed influence from up to 200 tokens of context. similar effects are reported by @xcite for neural dialogue models, and @xcite describe an alternative procedure for ablating contexts. context in transformer lms transformers introduce yet another mechanism for extracting information from long-range context: attention. attention is also used with rnns, but typically with just a single head-the hidden state still carries most of the information. in transformers, context enters into predictions primarily via unbounded random access. these models appear to benefit from significantly longer contexts than previous models. some recent work that investigates the behavior of individual transformer attention heads @xcite . this work finds that certain attention heads are sensitive to things like word frequency, positional information, and certain syntactic phenomena. while extremely informative about the computational structures implemented by fixed models, these approaches do not necessarily reveal anything about usable information: indeed, patterns of attention do not necessarily correlate with model predictions @xcite . other related work our finding that finegrained ordering information contributes little usable information is consistent with @xcite 's finding that long-range contexts could be informatively summarized in fixed-sized vectors; our finding that most usable information is carried by nouns is consistent with earlier findings about both specialized neural architectures @xcite and discourse representations in feature-based models @xcite . our approach also shares similar motivations to information-theoretic work on probing @xcite , which uses related tools to interpret linguistic structure in lm representations rather than characterizing their effect on lm predictions. several recent papers have explored the effect of training-time and test-time ablations in models for other data analysis tasks: pham et al. (2020) find that shuffling experiments have a limited effect on the accuracy of models for natural language inference, while @xcite describe several experiments aimed at introducing usable information for several question answering and sentence understanding tasks. ## discussion we have investigated the extent to which transformer models can use structural and lexical information in long-range contexts for english language modeling. experiments demonstrated that this information is primarily contained in content words and local ordering statistics: ablations that remove other kinds of information from context have little effect on models' predictive accuracies. in contrast, retaining only information about document identity or named entities causes significant drops in predictive accuracy: the effectiveness of long contexts is not explained by the presence of topic or named entity information alone. crucial to obtaining these results was a measure of ablated usable information grounded in the accuracy of models trained and tested on ablated contexts. past work on context in lms has primarily measured the influence of evaluation-time ablations. sometimes these two notions of contextsensitivity coincide (e.g., trigram shuffling) and sometimes they do not (e.g., removal of lexical information). our results also offer a jumping-off point for future modeling work. they motivate more efficient, compressed context representations that better preserve the information that is usable by current models. they motivate more accurate models by developing new context representations that make currently unusable information more prominent. several questions remain unanswered by our experiments. do ablations affect the quality of text generated by models? (in particular, does the usable information added by long contexts improve predictability of syntax, semantics, or simply document-level word frequency statistics?) more fundamentally, do observations about usable information reflect limitations of transformers or fundamental, (shannon-)information-theoretic properties of english? our results suggest that at least some of these effects are model-specific: deleting function words cannot add information, but improves held-out model accuracy. a complete answer to this question will require more detailed exploration, including a better understanding of human predictions in comparable settings.