How well does surprisal explain N400 amplitude under different experimental conditions? (2020)

Abstract: We investigate the extent to which word surprisal can be used to predict a neural measure of human language processing difficulty—the N400. To do this, we use recurrent neural networks to calculate the surprisal of stimuli from previously published neurolinguistic studies of the N400. We find that surprisal can predict N400 amplitude in a wide range of cases, and the cases where it cannot do so provide valuable insight into the neurocognitive processes underlying the response.

## introduction the n400 component of the event-related brain potential is generally understood to be a neural signal of processing difficulty @xcite . after over 1,000 articles published on the topic, we know that all else being equal, an upcoming word that is supported by the semantics of the context will elicit a lower-amplitude n400 than a word that is not @xcite . however, despite the great amount of experimental research on the topic, many aspects of the n400 are still not well understood. in addition to 'long-standing and recent linguistic [...] inputs' @xcite , the context that impacts n400 amplitude is thought to include factors such as world experience, attentional state, and mood @xcite . over the last decade, there have been a number of attempts to use computational modeling to test hypotheses about the neurocognitive processes underlying the n400 and how the aforementioned factors may impact its amplitude @xcite @xcite @xcite @xcite @xcite @xcite . as the majority of experimental research on the n400 involves manipulating the relationship between the stimulus and the preceding linguistic context @xcite , a computational account of how linguistic inputs impact n400 amplitude is a logical starting point. language models are inherently models of linguistic prediction based only on language input. since n400 amplitude reflects how unexpected an upcoming word is based on context, the predictions of a language model can be used to model how expected a word is based on the linguistic input, and thereby investigate the extent to which n400 amplitude is explainable by linguistic input alone. recent research has shown that surprisal, a measure of how unlikely a language model predicts the next word in sequence to be, correlates overall with n400 amplitude @xcite . thus, to investigate the extent to which n400 amplitude is explained by linguistic input alone, we ask to what extent surprisal can explain the variance observed in n400 amplitude. in order to investigate this, we run experimental stimuli from eleven experiments from six papers @xcite @xcite @xcite through two recurrent neural network language models @xcite , systematically comparing the significant predictors of n400 amplitude and surprisal. we find that in the majority of cases, significant differences in surprisal predict significant differences in n400 amplitude, and discuss the implications of the cases where it does not. ## approach, motivations, and hypotheses the aim of this study is to investigate the boundary conditions of using surprisal to model n400 amplitude. while there is evidence that surprisal and n400 amplitude are correlated overall @xcite , it is unclear what variance in n400 amplitude is actually being explained by surprisal. while it is tempting to assume that surprisal is correlated with the n400 because the same factors that lead to reduced n400 amplitudes lead to reduced surprisal, this has thus far not been shown empirically. this is the question that we investigate in this paper: which experimental manipulations that elicit a difference in n400 amplitude elicit the same difference in surprisal, and which do not? we do this by running the (english language) stimuli from previously published n400 studies through two neural networks that have been used extensively to model human language processing (e.g., in @xcite @xcite @xcite . the two models used are the the best english lstm from @xcite and big lstm+cnn inputs from @xcite , henceforth (following futrell et al., 2019) grnn and jrnn, respectively. these models are both lstm-rnn-lms, but differ most notably in size and training data: the jrnn has two hidden layers (8192 and 1024 units), a 793471word vocabulary, and was trained on 1 billion tokens @xcite ; while the grnn has two hidden layers (both 650 units), a 50000-word vocabulary, and was trained on 90 million tokens. in addition to answering questions about the nature of the neurocognitive systems underlying the n400, the results of this study also serve as a baseline for future research-they represent the best that current cognitively plausible neural network language models can do at predicting n400 amplitude using surprisal. thus, future research that argues for additional sources of information or neurocognitive processes being involved in the n400 on the basis of modeling success should demonstrate that the inclusion of such components in the model improves upon the results presented here. this aim of establishing a useful baseline is another reason for our choice of models-both are provided pre-trained by the authors, allowing for our results to be replicated and expanded upon. we also only use sets of stimuli that have been made available in papers or their supplementary materials. the stimuli from these papers @xcite @xcite @xcite , which cover a range of experimental manipulations that are discussed in section 4, are included in text format in our supplementary materials foot_0 . ## experiments figure 1 is a visualization of the findings of the original n400 studies and the results of the simulations. given the differences in measurements, there is no scale-the heights of the bars indicate which conditions elicited higher or lower n400 amplitudes or surprisals relative to the others in the same experiment or simulation. all and only the significant differences between conditions for significant predictors of the n400 or surprisal are shown, not including significant interactions with recording locations on the scalp (which are beyond the scope of the present study). black bars represent successful modeling of the differences in n400 amplitude, red bars represent unsuccessful or partially unsuccessful modeling, and purple bars indicate that the results are more complex than can be represented in this way. only stimuli sets with over 100 stimulus sentences were run through the models (grnn and jrnn); and while the models were not able to predict the surprisal of all target words (due to limited vocabularies or being unable to process certain characters in sentences), both models successfully calculated the surprisals of over 100 target words in each study. stimuli, target word surprisals, and the code used to run the models are all included in our supplementary materials. where possible, the significant predictors of the surprisal of the grnn and jrnn models were selected via backwards model selection using likelihood ratio tests of linear-mixed effects models (r core team, 2018; @xcite with and without the predictor under investigation as a main effect. when this was not possible, the significance of predictors were evaluated using a type iii anova with satterthwaite's method for estimating degrees of freedom @xcite . significant differences between experimental conditions (i.e. between the levels of a predictor) were t paper experiment jrnn surprisal grnn surprisal n400 amplitude a t a t a t a t a t a most few most few most few t a t a t a urbach & kutas (2010) exp. 1 urbach & kutas (2010) exp. 2 urbach & kutas (2010) exp. 2 urbach & kutas (2010) exp. 3 r kutas (1993) exp. u bc r u bc r u bc fr ito et al. (2016) exp. 1 u sr p fr u sr p fr u sr p fr ito et al. (2016) exp. 2 u sr p fr u sr p fr u sr p mm osterhout & mobley (1995) pronouns: target word m mm m mm m mm osterhout & mobley (1995) pronouns: sentencefinal word m osterhout & mobley (1995) anomalous: target word osterhout & mobley (1995) anomalous: sentence-final word syn ainsworth-darnell et al. (1998) exp. 1 da sem c syn da sem c sem da syn c ac @xcite exp. 1 av pc av @xcite exp. 2 nv pc mm m gramm. ## general discussion we compared human n400 responses with surprisal in two rnn-lms presented with the same stimuli, in the interest of determining the extent to which exposure to linguistic input alone can account for this particular component of human language processing. the results confirmed previous findings that surprisal is generally a good predictor of n400 amplitude, while also clearly demonstrating limitations of the models at capturing the human behavior. ## conclusions previous work has found that surprisal is a good predictor of n400 amplitude overall. comparisons of surprisal in rnn-lms to human n400 responses to the same input sentences showed for the first time that suprisal manages to account for a wide range of phenomena found in human n400 experiments. but at the same time, there are linguistic phenomena where it overpredicts, and others where it underpredicts a significant difference in the human n400 response. from the perspective of human language processing, this suggests that the activation of semantic and lexical features indexed by the n400 cannot be entirely captured by exposure to linguistic input alone. specifically, quantification, aspects of event structure, and morphosyntactic anomalies seem to require some other learning architecture than the bottom-up statistical learning represented by standard recurrent neural networks. from the perspective of model-building, in order to improve a language-model based cognitive model of the n400, we need to allow for the addition of more shallow semantic processing (independent of syntax and event structure) such as an implementation of spreading activation.