How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction (2021)

Abstract: After a neural sequence model encounters an unexpected token, can its behavior be predicted? We show that RNN and transformer language models exhibit structured, consistent generalization in out-of-distribution contexts. We begin by introducing two idealized models of generalization in next-word prediction: a lexical context model in which generalization is consistent with the last word observed, and a syntactic context model in which generalization is consistent with the global structure of the input. In experiments in English, Finnish, Mandarin, and random regular languages, we demonstrate that neural language models interpolate between these two forms of generalization: their predictions are well-approximated by a log-linear combination of lexical and syntactic predictive distributions. We then show that, in some languages, noise mediates the two forms of generalization: noise applied to input tokens encourages syntactic generalization, while noise in history representations encourages lexical generalization. Finally, we offer a preliminary theoretical explanation of these results by proving that the observed interpolation behavior is expected in log-linear models with a particular feature correlation structure. These results help explain the effectiveness of two popular regularization schemes and show that aspects of sequence model generalization can be understood and controlled.

## introduction neural language models (lms) play a key role in language processing systems for tasks as diverse as machine translation, dialogue, and automated speech recognition @xcite @xcite . these lms, which model distributions over words in context via recurrent, convolutional, or attentional neural networks, have been found to consistently outperform finite-state approaches to language modeling based on hidden markov models @xcite neural language model global context x g local ctx. x l next token p lm (x n )? the pizza ate cat tasted corpus data p(x 1:n ) regular language p(x 1:n ) a c b w x y a c x ? x w y x? y? w? the? delicious? global generalization: ## models of generalization consider the example contexts shown in (a-c). each is an extremely unlikely sentence prefix, featuring text that is globally inconsistent with english syntax or semantic constraints. in such contexts, is it possible to predict a priori what a neural lm trained on language data will do next? we can formalize the situation depicted in these examples as follows: let p(x 1:n ) = p(x 1 , x 2 , . . . , x n ) be a distribution over sentences with tokens x i , and let in the remainder of this section, we describe a set of candidate hypotheses about what this next-token distribution might look like, and in section 4 evaluate the extent to which these hypotheses accurately predict the true behavior of p lm . ## experiments which of these models (if any) best describes the empirical behavior of neural lms trained on real datasets? in this section, we present two sets of evaluations. the first aims to characterize how well pl , pg , and combinations of the two predict the out-of-distribution behavior of @xcite and transformer @xcite language models with standard training. the second explores whether these training procedures can be modified to control the relative strength of local and global generalization. both sets of experiments investigate the behavior of rnn and transformer lms on a diverse set of datasets: first, a collection of random regular languages in which the true data distribution p(x) can be precisely modeled; second, a collection of natural language datasets from three languages (mandarin chinese, english, and finnish) which vary in the flexibility of their word order and the complexity of their morphology. we begin with a more detailed discussion of models and datasets in section 4.1; then describe generalization experiments in section 4.2 and control experiments in section 4.3. ## explaining the experiments with empirical evidence that interpolation between local and global models is a good approximation to out-of-distribution language model behavior, we next investigate whether this behavior can be explained theoretically. while we leave for future work a complete answer to this question, we conclude with the following proposition, which describes a set of conditions under which lm generalization will be well approximated by log-linear interpolation between pl and pg . proposition 1. let θ be the parameters of a log-linear model optimizing: in other words, for log-linear language models with informative local and global features, the observed effectiveness of multiplicative interpolation is expected. proof is given in appendix c. it is important to qualify this result in several ways: it relies on a feature function φ that may not be a realistic representation of the context feature produced by deep network models, involves strong assumptions about the independent predictive power of local and global features at training time, and becomes vacuous for large values of or small values of λ. it is weak in absolute sense (e 4 /λ grows considerably faster than except for very large values of λ); its function is simply to relate predictions in surprising contexts to measurable properties of the training distribution. nevertheless, the result shows that some aspects of interpolation behavior can be predicted from the parametric form of predictors alone; future work might strengthen this claim to more directly characterize the neural network predictors studied in this paper and explain the observed differences across languages. ## conclusion when neural sequence models are exposed to outof-distribution contexts with conflicting local and global information, their behavior can be predicted. across natural and synthetic data distribution, sequence model generalization appears to be well approximated by either a local (n-gram-like) or global (skip-gram-like) predictor, and best approximated by a log-linear interpolation of the two whose weight can sometimes be controlled by noise-based regularization. this work suggests several avenues for future exploration: first, explaining data-dependent aspects of the local-global tradeoff (especially cross-linguistic differences that are not clearly explained by typological differences between languages); second, determining whether architectural improvements to standard sequence models can even more effectively target specific kinds of structured generalization. for any θ v,i we have: we can then obtain the main result: proof of proposition 1. first note that we can estimate both local and global models using distributions of the form: here we have used θ to denote the parameters of the full model, η to denote the parameters of the global-only model, and µ to denote the parameters of the local-only model. because each φ has an indicator for local, global, and (local, global) values, only two features are active in surprising contexts, corresponding to x g and x l . we will denote the indices of these weights i and j in each weight vector: without loss of generality, assume the first term is larger than the second. by applying lemma 1, we can rewrite θ in terms of η and µ. for shorthand, we will write ∆ = /λ: note that this proof is not specific to local and global feature representations, but applies generically to any pair of regularized log-linear models with overlapping features and similar predictions. the proof is adapted from theorem 1 of @xcite , and we expect that it could be strengthened (as done there) to depend only on correlations between features of φ(x g , x l ) and each of {φ(x g ), φ(x l )}.