Original: The more human-like the language model, the more surprisal is the best predictor of N400 amplitude
Revised: The more human-like the language model, the more surprisal is the best predictor of N400 amplitude

Original: Abstract
Revised: Abstract

Original: Under information-theoretic accounts of language comprehension, the effort required to process a word is correlated with surprisal, the negative log-probabilityof that word given its context.
Revised: Under information-theoretic accounts of language comprehension, the effort required to process a word is correlated with surprisal, the negative log-probability of that word given its context.

Original: This can (equivalently) be considered to reflectcognitive effort in proportion to the amount of information conveyed by a givenword (Frank et al., 2015), or the amount of effort required to update the our incremental predictions about upcoming words (Levy, 2008; Aurnhammer and Frank,2019).
Revised: This can (equivalently) be considered to reflect cognitive effort in proportion to the amount of information conveyed by a given word (Frank et al., 2015), or the amount of effort required to update the our incremental predictions about upcoming words (Levy, 2008; Aurnhammer and Frank, 2019).

Original: In contrast, others (e.g. Brothers and Kuperberg, 2021) have argued thatprocessing difficulty is proportional to the contextual probability of a word, thuspositing a linear (rather than logarithmic) relationship between word probabilityand processing difficulty.
Revised: In contrast, others (e.g. Brothers and Kuperberg, 2021) have argued that processing difficulty is proportional to the contextual probability of a word, thus positing a linear (rather than logarithmic) relationship between word probability and processing difficulty.

Original: We investigate which of these two accounts best explainthe N400, a neural response that provides some of the best evidence for predictionin language comprehension (Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020).
Revised: We investigate which of these two accounts best explain the N400, a neural response that provides some of the best evidence for prediction in language comprehension (Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020).

Original: To do this, we expand upon previous work by comparing howwell the probability and surprisal calculated by 43 transformer language modelspredict N400 amplitude.
Revised: To do this, we expand upon previous work by comparing how well the probability and surprisal calculated by 43 transformer language models predict N400 amplitude.

Original: We thus investigate both which models’ predictions bestpredict the N400, and for each model, whether surprisal or probability is moreclosely correlated with N400 amplitude.
Revised: We thus investigate both which models’ predictions best predict the N400, and for each model, whether surprisal or probability is more closely correlated with N400 amplitude.

Original: We find that of the models tested, OPT6.7B and GPT-J are reliably the best at predicting N400 amplitude, and that forthese transformers, surprisal is the better predictor.
Revised: We find that of the models tested, OPT6.7B and GPT-J are reliably the best at predicting N400 amplitude, and that for these transformers, surprisal is the better predictor.

Original: In fact, we find that the morehighly correlated the predictions of a language model are with N400 amplitude,the greater the extent to which surprisal is a better predictor than probability.
Revised: In fact, we find that the more highly correlated the predictions of a language model are with N400 amplitude, the greater the extent to which surprisal is a better predictor than probability.

Original: Sincelanguage models that more closely mirror human statistical knowledge are morelikely to be informative about the human predictive system, these results supportthe information-theoretic account of language comprehension.
Revised: Since language models that more closely mirror human statistical knowledge are more likely to be informative about the human predictive system, these results support the information-theoretic account of language comprehension.

Original:  Introduction
Revised:  Introduction

Original: If the language comprehension system follows information-theoretic principles, we should expectthat the information content conveyed by a word should play a crucial role in how that word isprocessed.
Revised: If the language comprehension system follows information-theoretic principles, we should expect that the information content conveyed by a word should play a crucial role in how that word is processed.

Original: This is a core feature of surprisal theory : the idea that the difficulty in processing aword in context is correlated with the surprisal of that word, that is, the negative logarithm of theprobability of the word in context (Hale, 2001, 2003; Levy, 2008; Smith and Levy, 2011, 2013; Franket al., 2015; Aurnhammer and Frank, 2019; Michaelov and Bergen, 2020).
Revised: This is a core feature of surprisal theory : the idea that the difficulty in processing a word in context is correlated with the surprisal of that word, that is, the negative logarithm of the probability of the word in context (Hale, 2001, 2003; Levy, 2008; Smith and Levy, 2011, 2013; Frank et al., 2015; Aurnhammer and Frank, 2019; Michaelov and Bergen, 2020).

Original: A key mathematicalproperty of lexical surprisal is that it reflects the information content of a word (as formally definedinformation-theoretically), but also, as noted by Levy (2008), is equivalent to the Kullback-Leiblerdivergence (Kullback and Leibler, 1951) between the predicted probability distribution for the nextword and the true probability distribution (1 for the actual word and 0 for all other possible words).
Revised: A key mathematical property of lexical surprisal is that it reflects the information content of a word (as formally defined information-theoretically), but also, as noted by Levy (2008), is equivalent to the Kullback-Leibler divergence (Kullback and Leibler, 1951) between the predicted probability distribution for the next word and the true probability distribution (1 for the actual word and 0 for all other possible words).

Original: Thus, surprisal as a metric can be thought to account not only for the increased effort required to process a word with high information content (Hale, 2001; Frank et al., 2015), but also the effortrequired to update the entire predicted probability distribution based on encountering a word.
Revised: Thus, surprisal as a metric can be thought to account not only for the increased effort required to process a word with high information content (Hale, 2001; Frank et al., 2015), but also the effort required to update the entire predicted probability distribution based on encountering a word.

Original: An alternative to surprisal theory is what Brothers and Kuperberg (2021) term the proportionalpreactivation account , which also posits that prediction plays a key role in language comprehension.
Revised: An alternative to surprisal theory is what Brothers and Kuperberg (2021) term the proportional preactivation account , which also posits that prediction plays a key role in language comprehension.

Original: However, instead of processing difficulty reflecting the surprisal of a word in context, it reflects theextent to which the word was not predicted—that is, instead of − log ( p ) , scales with 1 − p .
Revised: However, instead of processing difficulty reflecting the surprisal of a word in context, it reflects the extent to which the word was not predicted—that is, instead of − log ( p ) , scales with 1 − p .

Original: Thus,under surprisal theory, the mathematical relationship between lexical probability and processingdifficulty is logarithmic, while under the proportional preactivation account, it is linear.
Revised: Thus, under surprisal theory, the mathematical relationship between lexical probability and processing difficulty is logarithmic, while under the proportional preactivation account, it is linear.

Original: To measure real-time human language processing, we turn to the N400, a neural response that offersa snapshot into the neurocognitive system underlying language comprehension.
Revised: To measure real-time human language processing, we turn to the N400, a neural response that offers a snapshot into the neurocognitive system underlying language comprehension.

Original: It is widely agreedthat the N400 reflects the brain activation driven by encountering a word and that it is modulatedby the extent to which the word (or its semantic features) are preactivated by the preceding context(Kutas and Federmeier, 2011; Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020;
Revised: It is widely agreed that the N400 reflects the brain activation driven by encountering a word and that it is modulated by the extent to which the word (or its semantic features) are preactivated by the preceding context (Kutas and Federmeier, 2011; Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020;

Original: Federmeier, 2021).
Revised: Federmeier, 2021).

Original: Thus, the N400 is perhaps the most suitable processing difficulty metric forevaluating the predictions of surprisal theory (Michaelov and Bergen, 2020).
Revised: Thus, the N400 is perhaps the most suitable processing difficulty metric for evaluating the predictions of surprisal theory (Michaelov and Bergen, 2020).

Original: Despite this fact, tothe best of our knowledge, the question of whether surprisal or raw probability is a better predictorof N400 amplitude has only been tested for two language models thus far—Yan and Jaeger (2020)use a mixture of a 5-gram model and ‘skip bi-gram’ (both trained by Frank and Willems, 2017),and Szewczyk and Federmeier (2022) use GPT-2.
Revised: Despite this fact, to the best of our knowledge, the question of whether surprisal or raw probability is a better predictor of N400 amplitude has only been tested for two language models thus far—Yan and Jaeger (2020) use a mixture of a 5-gram model and ‘skip bi-gram’ (both trained by Frank and Willems, 2017), and Szewczyk and Federmeier (2022) use GPT-2.

Original: Both find that surprisal better fits the N400 data,though on one dataset, Szewczyk and Federmeier (2022) find that when restricting their analysis toonly expected (higher-probability) items, raw probability appears to be a better predictor; while whenrestricting their analysis to unexpected (lower-probability items), surprisal is the better predictor.
Revised: Both find that surprisal better fits the N400 data, though on one dataset, Szewczyk and Federmeier (2022) find that when restricting their analysis to only expected (higher-probability) items, raw probability appears to be a better predictor; while when restricting their analysis to unexpected (lower-probability items), surprisal is the better predictor.

Original: In the present study, we substantially expand upon this previous work.
Revised: In the present study, we substantially expand upon this previous work.

Original: We run the stimuli froma previously-published N400 study (Nieuwland et al., 2018) through 43 neural language models,calculate the probability and surprisal for each item with each model, and use these to predict N400amplitude.
Revised: We run the stimuli from a previously-published N400 study (Nieuwland et al., 2018) through 43 neural language models, calculate the probability and surprisal for each item with each model, and use these to predict N amplitude.

Original: First, we ask which model is the best predictor of N400 amplitude, and whether surprisalor raw probability is a better predictor in all models.
Revised: First, we ask which model is the best predictor of N400 amplitude, and whether surprisal or raw probability is a better predictor in all models.

Original: Additionally, because models that behave moresimilarly to humans overall are also more likely to be more informative about the human predictivesystem, we also compare surprisal and probability across models as a function of how well theypredict the human N400 response overall.
Revised: Additionally, because models that behave more similarly to humans overall are also more likely to be more informative about the human predictive system, we also compare surprisal and probability across models as a function of how well they predict the human N400 response overall.

Original:  Method
Revised:  Method

Original: Following research showing that N400 amplitude is best predicted by transformers (Merkx and Frank,2021; Michaelov et al., 2022), we restricted our analysis to contemporary transformer languagemodels made available through the transformers (Wolf et al., 2020)
Revised: Following research showing that N400 amplitude is best predicted by transformers (Merkx and Frank, 2021; Michaelov et al., 2022), we restricted our analysis to contemporary transformer language models made available through the transformers (Wolf et al., 2020)

Original: Python package.
Revised: Python package.

Original: These included43 models of the GPT-2 (Radford et al., 2019), GPT-Neo (Black et al., 2021), GPT-J (Wang and
Revised: These included 43 models of the GPT-2 (Radford et al., 2019), GPT-Neo (Black et al., 2021), GPT-J (Wang and

Original: Komatsuzaki, 2021), OPT (Zhang et al., 2022), (Devlin et al., 2019), RoBERTa (Liu et al., 2019),
Revised: Komatsuzaki, 2021), OPT (Zhang et al., 2022), (Devlin et al., 2019), RoBERTa (Liu et al., 2019),

Original: ALBERT (Lan et al., 2020), XGLM (Lin et al., 2021), BLOOM (BigScience, 2022), Multilingual
Revised: ALBERT (Lan et al., 2020), XGLM (Lin et al., 2021), BLOOM (BigScience, 2022), Multilingual

Original: BERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), and XLM-R (Conneau et al., 2020;
Revised: BERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), and XLM-R (Conneau et al., 2020;

Original: Goyal et al., 2021) architectures.
Revised: Goyal et al., 2021) architectures.

Original: All models used are shown in Figure 1.
Revised: All models used are shown in Figure 1.

Original: For our N400 data, we used a subset of the data from an experiment carried out by (Nieuwland et al.,2018), a large-scale study with 334 participants.
Revised: For our N400 data, we used a subset of the data from an experiment carried out by (Nieuwland et al., 2018), a large-scale study with 334 participants.

Original: Specifically, we look at the N400 amplitudes elicitedby nouns that are either more or less contextually predictable.
Revised: Specifically, we look at the N400 amplitudes elicited by nouns that are either more or less contextually predictable.

Original: Nieuwland et al.
Revised: Nieuwland et al.

Original: (2018) operationalize
Revised: (2018) operationalize

Original: N400 amplitude as mean voltage between 200-500ms after stimulus presentation at 6 electrodes intheir region of interest (Cz, C3, C4, Pz, P3, and P4).
Revised: N400 amplitude as mean voltage between 200-500ms after stimulus presentation at 6 electrodes in their region of interest (Cz, C3, C4, Pz, P3, and P4).

Original: To investigate how well the language models’ predictions correlate with N400 amplitude, we raneach of the 160 stimulus sentences from the Nieuwland et al.
Revised: To investigate how well the language models’ predictions correlate with N400 amplitude, we ran each of the 160 stimulus sentences from the Nieuwland et al.

Original: (2018) study up until the critical nounthrough each of the the 43 language models, calculating the probability and surprisal of each of thecritical nouns.
Revised: (2018) study up until the critical noun through each of the the 43 language models, calculating the probability and surprisal of each of the critical nouns.

Original: Not all critical nouns were in each model’s vocabulary.
Revised: Not all critical nouns were in each model’s vocabulary.

Original: The fairest approach is tocompare only words appearing in the vocabulary of all models (see Michaelov and Bergen, 2020;
Revised: The fairest approach is to compare only words appearing in the vocabulary of all models (see Michaelov and Bergen, 2020;

Original: Michaelov et al., 2022 for similar work following this approach).
Revised: Michaelov et al., 2022 for similar work following this approach).

Original: However, with stimuli in this study,this means that only data from 37 sentences could be analyzed.
Revised: However, with stimuli in this study, this means that only data from 37 sentences could be analyzed.

Original: An alternative approach is to predicteach sub-word token of the critical noun given the preceding context and the preceding sub-wordtokens, and then either taking the product of probabilities or sum of the surprisals to get an overallword probability or surprisal. This produces a probability and surprisal for each model; however,
Revised: An alternative approach is to predict each sub-word token of the critical noun given the preceding context and the preceding sub-word tokens, and then either taking the product of probabilities or sum of the surprisals to get an overall

Original: b i g
Revised: b i g

Original: sc i en c e − b l oo m − 7b b
Revised: sc i en c e − b l oo m − 7b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 3b x
Revised: sc i en c e − b l oo m − 3b x

Original: l m − m l m − 17 −b
Revised: l m − m l m − 17 −b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b b
Revised: sc i en c e − b l oo m − 1b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b a l be r t − ba s e − vbe r t − l a r ge − un c a s ed
Revised: sc i en c e − b l oo m − 1b a l be r t − ba s e − vbe r t − l a r ge − un c a s ed

Original: G eo t r end − be r t − ba s e − 15 l ang − c a s ed be r t − ba s e − m u l t ili ngua l − c a s ed
Revised: G eo t r end − be r t − ba s e − 15 l ang − c a s ed be r t − ba s e − m u l t ili ngua l − c a s ed

Original: G eo t r end − be r t − ba s e − 10 l ang − c a s ed be r t − ba s e − c a s ed be r t − ba s e − un c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e b
Revised: G eo t r end − be r t − ba s e − 10 l ang − c a s ed be r t − ba s e − c a s ed be r t − ba s e − un c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 560 m be r t − l a r ge − c a s ed f a c eboo k − op t −m x l m − m l m − 100 − f a c eboo k − op t −m f a c eboo
Revised: sc i en c e − b l oo m − 560 m be r t − l a r ge − c a s ed f a c eboo k − op t −m x l m − m l m − 100 − f a c eboo k − op t −m f a c eboo

Original: k − x g l m − 564 M gp tgp t 2 − m ed i u m
Revised: k − x g l m − 564 M gp tgp t 2 − m ed i u m

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo − 125 M x l m − r obe r t a − l a r ge
Revised: I − gp t − neo − 125 M x l m − r obe r t a − l a r ge

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo −r obe r t a − ba s e f a c eboo k − x g l m −be r t − l
Revised: I − gp t − neo −r obe r t a − ba s e f a c eboo k − x g l m −be r t − l

Original: a r ge − un c a s ed − w ho l e − w o r d − m a sk i ng be r t − l a r ge − c a s ed − w ho l e − w o r d − m a
Revised: a r ge − un c a s ed − w ho l e − w o r d − m a sk i ng be r t − l a r ge − c a s ed − w ho l e − w o r d − m a

Original: sk
Revised: sk

Original: i ng f a c eboo k − op t
Revised: i ng f a c eboo k − op t

Original: −f a c eboo k − x g l m −f a c eboo k − x g l m −f a c eboo k − op t −
Revised: −f a c eboo k − x g l m −f a c eboo k − x g l m −f a c eboo k − op t −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo − 2 gp t 2 − x l gp t 2 − l a r ge a l be r t − x l a r ge − vf a c eboo k − x g l m −a l be r t − xx l a r ge − vr obe r t a − l a r ge f a c eboo k − op t −
Revised: I − gp t − neo − 2 gp t 2 − x l gp t 2 − l a r ge a l be r t − x l a r ge − vf a c eboo k − x g l m −a l be r t − xx l a r ge − vr obe r t a − l a r ge f a c eboo k − op t −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − j − 6 B f a c eboo k − x l m − r obe r t a − x l
Revised: I − gp t − j − 6 B f a c eboo k − x l m − r obe r t a − x l

Original: Model
Revised: Model

Original: x l m − m l m − 17 −be r t − ba s e − c a s ed be r t − l a r ge − c a s ed be r t − ba s e − un c a s ed be r t − l
Revised: x l m − m l m − 17 −be r t − ba s e − c a s ed be r t − l a r ge − c a s ed be r t − ba s e − un c a s ed be r t − l

Original: a r ge − un c a s ed b
Revised: a r ge − un c a s ed b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b b
Revised: sc i en c e − b l oo m − 1b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 7b x
Revised: sc i en c e − b l oo m − 7b x

Original: l m − m l m − 100 − b
Revised: l m − m l m − 100 − b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b b
Revised: sc i en c e − b l oo m − 1b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 560 m
Revised: sc i en c e − b l oo m − 560 m

Original: b
Revised: b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 3b be r t − ba s e − m u l t ili ngua l − c a s ed
Revised: sc i en c e − b l oo m − 3b be r t − ba s e − m u l t ili ngua l − c a s ed

Original: G eo t r end − be r t − ba s e − 15 l ang − c a s ed a l be r t − ba s e − v
Revised: G eo t r end − be r t − ba s e − 15 l ang − c a s ed a l be r t − ba s e − v

Original: G eo t r end − be r t − ba s e − 10 l ang − c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e be r t − l a r ge − c a s ed − w ho l e − w o r d − m a
Revised: G eo t r end − be r t − ba s e − 10 l ang − c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e be r t − l a r ge − c a s ed − w ho l e − w o r d − m a

Original: sk i ng a l be r t − x l a r ge − vx l m − r obe r t a − l a r ge f a c eboo k − x l m − r obe r t a − x l be r t − l a r ge − un c a s ed − w ho l e − w o r d − m a
Revised: sk i ng a l be r t − x l a r ge − vx l m − r obe r t a − l a r ge f a c eboo k − x l m − r obe r t a − x l be r t − l a r ge − un c a s ed − w ho l e − w o r d − m a

Original: sk i ng a l be r t − xx l a r ge − vf a c eboo
Revised: sk i ng a l be r t − xx l a r ge − vf a c eboo

Original: k − x g l m − 564 M r obe r t a − ba s e
Revised: k − x g l m − 564 M r obe r t a − ba s e

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo − 125 M gp tf a c eboo k − op t −m f a c eboo k − op t −m r obe r t a − l a r ge f a c eboo k − x g l m −f a c eboo k − x g l m −gp t 2 − m ed i u m gp t 2 − x l gp t 2 − l a r ge f a c eboo k − x g l m −
Revised: I − gp t − neo − 125 M gp tf a c eboo k − op t −m f a c eboo k − op t −m r obe r t a − l a r ge f a c eboo k − x g l m −f a c eboo k − x g l m −gp t 2 − m ed i u m gp t 2 − x l gp t 2 − l a r ge f a c eboo k − x g l m −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo −f a c eboo k − x g l m −
Revised: I − gp t − neo −f a c eboo k − x g l m −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo −f a c eboo k − op t −f a c eboo k − op t −f a c eboo k − op t −
Revised: I − gp t − neo −f a c eboo k − op t −f a c eboo k − op t −f a c eboo k − op t −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − j − 6 B
Revised: I − gp t − j − 6 B

Original: Model
Revised: Model

Original: All
Revised: All

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Model
Revised: Model

Original: [Table]
Revised: [Table]

Original: Probability
Revised: Probability

Original: Surprisal  given the differences in tokenization (including tokenizer algorithm) and training tasks (for example,traditional vs. masked language modeling, and the possible addition of whole-word masking to thelatter task), this may introduce possible confounds.
Revised: Surprisal word probability or surprisal. This produces a probability and surprisal for each model; however, given the differences in tokenization (including tokenizer algorithm) and training tasks (for example, traditional vs. masked language modeling, and the possible addition of whole-word masking to the latter task), this may introduce possible confounds.

Original: Thus, in our analyses, we try both approaches.
Revised: Thus, in our analyses, we try both approaches.

Original: Probability, surprisal, and N400 amplitude were all z-scored.
Revised: Probability, surprisal, and N400 amplitude were all z-scored.

Original:  Results 
Revised:  Results 3.1 Single-token critical nouns

Original: 3.1 Single-token critical nouns

Original: First, we run our analyses for only the critical nouns that are single tokens in all models.
Revised: First, we run our analyses for only the critical nouns that are single tokens in all models.

Original: Weconstructed linear mixed-effects regressions following (Michaelov et al., 2022), with the relevantlanguage model probability or surprisal as a main effect, with the laboratory that the experiment wascarried out in as a covariate, and random intercepts of experimental subject and stimulus item.
Revised: We constructed linear mixed-effects regressions following (Michaelov et al., 2022), with the relevant language model probability or surprisal as a main effect, with the laboratory that the experiment was carried out in as a covariate, and random intercepts of experimental subject and stimulus item.

Original: Wethen calculated each regression’s Akaike Information Criterion (AIC; Akaike, 1973) as a metric of fit.
Revised: We then calculated each regression’s Akaike Information Criterion (AIC; Akaike, 1973) as a metric of fit.

Original: We then compared regression fit across model and metric.
Revised: We then compared regression fit across model and metric.

Original: The results can be seen in Figure 1A.
Revised: The results can be seen in Figure 1A.

Original: First, in line with previous work, for the models that can be used to best predict N400 amplitude—
Revised: First, in line with previous work, for the models that can be used to best predict N400 amplitude—

Original: XLM-R XL , GPT-J, OPT-6B,
Revised: XLM-R XL , GPT-J, OPT-6B,

Original: etc.—surprisal better predicts N400 amplitude than raw probability.
Revised: etc.—surprisal better predicts N400 amplitude than raw probability.

Original: However, in contrast to previous work, probability is a better predictor for some other models.
Revised: However, in contrast to previous work, probability is a better predictor for some other models.

Original: Totest for trends across all models, we ran a likelihood ratio test using linear mixed-effects regressions.
Revised: To test for trends across all models, we ran a likelihood ratio test using linear mixed-effects regressions.

Original: AIC was the dependent variable, and we compared a null model with only language model as randomintercept with one also including metric (either probability or surprisal) as a fixed effect, findingno significant difference ( χ 2 (1) = 3 . 62 , p = 0 . 0572 ).
Revised: AIC was the dependent variable, and we compared a null model with only language model as random intercept with one also including metric (either probability or surprisal) as a fixed effect, finding no significant difference ( χ 2 (1) = 3 . 62 , p = 0 . 0572 ).

Original: Thus, there is no overall trend as to whethersurprisal or probability is a better predictor of N400 amplitude.
Revised: Thus, there is no overall trend as to whether surprisal or probability is a better predictor of N400 amplitude.

Original: However, as can be seen in the graph, it appears that for models whose predictions are better predictorsof N400 amplitude, surprisal tends to be a better predictor of N400 amplitude than probability.
Revised: However, as can be seen in the graph, it appears that for models whose predictions are better predictors of N400 amplitude, surprisal tends to be a better predictor of N400 amplitude than probability.

Original: Inother words, it appears that as model performance at N400 amplitude prediction improves, the extentto which surprisal is a better predictor than probability increases.
Revised: In other words, it appears that as model performance at N400 amplitude prediction improves, the extent to which surprisal is a better predictor than probability increases.

Original: To test this, we constructed asimple least-squares regression with the surprisal regression AICs as the independent variable and thedifference between surprisal and probability regression AICs as the dependent variable.
Revised: To test this, we constructed a simple least-squares regression with the surprisal regression AICs as the independent variable and the difference between surprisal and probability regression AICs as the dependent variable.

Original: We found asignificant correlation between the two ( F (1 , 41) = 11 .
Revised: We found a significant correlation between the two ( F (1 , 41) = 11 .

Original: 06 , p = 0 . 0019 )
Revised: 06 , p = 0 . 0019 )

Original: in the previously-discusseddirection—as surprisal regression AIC decreases, the extent to which surprisal regression AIC islower than probability regression AIC increases.
Revised: in the previously-discussed direction—as surprisal regression AIC decreases, the extent to which surprisal regression AIC is lower than probability regression AIC increases.

Original: We also constructed a least-squares regression inthe same vein but with probability regression AIC as the independent variable; which also displayeda significant correlation ( F (1 , 41) = 90 .
Revised: We also constructed a least-squares regression in the same vein but with probability regression AIC as the independent variable; which also displayed a significant correlation ( F (1 , 41) = 90 .

Original: 15 , p = 0 < 0 . 0001 ).
Revised: 15 , p = 0 < 0 . 0001 ).

Original: Thus, we find that as language modelperformance at N400 prediction (whether based on surprisal or raw probability) increases, so doesthe extent to which surprisal is a better predictor.
Revised: Thus, we find that as language model performance at N400 prediction (whether based on surprisal or raw probability) increases, so does the extent to which surprisal is a better predictor.

Original: 3.2 All critical words
Revised: 3.2 All critical words

Original: We then ran the same analyses on all the stimuli, where the critical nouns were either one or moretokens in all models.
Revised: We then ran the same analyses on all the stimuli, where the critical nouns were either one or more tokens in all models.

Original: As before, we constructed linear mixed-effects regressions in the same way,using AIC as a metric of fit.
Revised: As before, we constructed linear mixed-effects regressions in the same way, using AIC as a metric of fit.

Original: The results are shown inFigure 1B.
Revised: The results are shown inFigure 1B.

Original: On this analysis, the ranking of which models best predict N400 amplitude differs.
Revised: On this analysis, the ranking of which models best predict N400 amplitude differs.

Original: But the higher-levelpatterns do not: there is no significant difference in performance between surprisal and probabilityoverall ( χ 2 (1) = 0 .
Revised: But the higher-level patterns do not: there is no significant difference in performance between surprisal and probability overall ( χ 2 (1) = 0 .

Original: 97 , p = 0 . 3258 ), and regression AIC ( F (1 , 41)
Revised: 97 , p = 0 . 3258 ), and regression AIC ( F (1 , 41)

Original: = 218 , p < 0 . 0001 ) andprobability regression AIC ( F (1 , 41) = 30 .
Revised: = 218 , p < 0 . 0001 ) and probability regression AIC ( F (1 , 41) = 30 .

Original: 6 , p < 0 . 0001 ) are correlated with the difference betweenthe two in the same direction—that is, as the extent to which the predictions of a language modelpredict N400 amplitude improves, so does the extent to which surprisal is a better predictor.
Revised: 6 , p < 0 . 0001 ) are correlated with the difference between the two in the same direction—that is, as the extent to which the predictions of a language model predict N400 amplitude improves, so does the extent to which surprisal is a better predictor.

Original:  Discussion
Revised:  Discussion

Original: Clear patterns are seen in both sets of results.
Revised: Clear patterns are seen in both sets of results.

Original: Under both analyses, neither surprisal nor probability isbetter overall, but for the models that best predict N400 amplitude (when either surprisal or probabilityis considered), surprisal is the best predictor.
Revised: Under both analyses, neither surprisal nor probability is better overall, but for the models that best predict N400 amplitude (when either surprisal or probability is considered), surprisal is the best predictor.

Original: This in fact is shown to be a statistically significantregularity: and as the performance of language model predictions at predicting N400 amplitudeimproves, the extent to which surprisal is a better predictor than probability increases.
Revised: This in fact is shown to be a statistically significant regularity: and as the performance of language model predictions at predicting N400 amplitude improves, the extent to which surprisal is a better predictor than probability increases.

Original: The question of which model best predicts N400 amplitude is more complicated.
Revised: The question of which model best predicts N400 amplitude is more complicated.

Original: Masked languagemodels appear to perform worse at modeling N400 amplitude when all tokens are consideredcompared to the when only single-token words are considered.
Revised: Masked language models appear to perform worse at modeling N400 amplitude when all tokens are considered compared to the when only single-token words are considered.

Original: This is striking given that under thesingle-token-only analysis, the best-performing model (XLM-R XL ) is a masked language model.
Revised: This is striking given that under the single-token-only analysis, the best-performing model (XLM-R XL ) is a masked language model.

Original: Thispattern is likely at least partly explained by training task.
Revised: This pattern is likely at least partly explained by training task.

Original: Specifically, predicting later sub-wordtokens based only on their preceding context and preceding sub-word tokens occurs with everysingle multi-token word for autoregressive language models, but for masked language models, thisis not necessarily the case.
Revised: Specifically, predicting later sub-word tokens based only on their preceding context and preceding sub-word tokens occurs with every single multi-token word for autoregressive language models, but for masked language models, this is not necessarily the case.

Original: On the one hand, this suggests that if attempting to test which modelbest predicts a human comprehension metric like the N400, it may be best to use single-token-onlyanalyses.
Revised: On the one hand, this suggests that if attempting to test which model best predicts a human comprehension metric like the N400, it may be best to use single-token-only analyses.

Original: On the other hand, however, from a cognitive modeling perspective, it is valuable to beable to analyze all stimuli, especially if the surprisal of a sequence of two or more words is needed.
Revised: On the other hand, however, from a cognitive modeling perspective, it is valuable to be able to analyze all stimuli, especially if the surprisal of a sequence of two or more words is needed.

Original: Our results suggest that for such experiments, autoregressive models such as GPT-J and OPT-6B (thebest-performing in our experiment) should be used.
Revised: Our results suggest that for such experiments, autoregressive models such as GPT-J and OPT-6B (the best-performing in our experiment) should be used.

Original: We now return to the question of how well an information-theoretic account of language processing,surprisal theory, accounts for the experimental neurocognitive data, compared to the proportionalpreactivation account.
Revised: We now return to the question of how well an information-theoretic account of language processing, surprisal theory, accounts for the experimental neurocognitive data, compared to the proportional preactivation account.

Original: The results show that for models with more human-like predictions, surprisalbetter fits the data .
Revised: The results show that for models with more human-like predictions, surprisal better fits the data.

Original: Because these models are more likely to also reveal the mathematical relationshipbetween lexical statistics and human predictive processing, this result supports the informationtheoretic account.
Revised: Because these models are more likely to also reveal the mathematical relationship between lexical statistics and human predictive processing, this result supports the informationtheoretic account.

Original: Finally, we may want to ask why it is the case that for less human-like models, probability bettermodels N400 amplitude.
Revised: Finally, we may want to ask why it is the case that for less human-like models, probability better models N400 amplitude.

Original: One possible explanation relates to Szewczyk and Federmeier’s (2022)finding that language model probability is more closely correlated with N400 amplitude for higherprobability items, while surprisal is more closely correlated for lower-probability items.
Revised: One possible explanation relates to Szewczyk and Federmeier’s (2022) finding that language model probability is more closely correlated with N400 amplitude for higherprobability items, while surprisal is more closely correlated for lower-probability items.

Original: Becausesurprisal is a logarithmic metric, small differences at the extremely low-probability end of the scaleare magnified.
Revised: Because surprisal is a logarithmic metric, small differences at the extremely low-probability end of the scale are magnified.

Original: Thus, while there may only be small differences in probability for such items, formodels whose predictions are less human-like, any divergence from human-like predictions aremagnified at this end of the scale; while this would not be the case with raw probability.
Revised: Thus, while there may only be small differences in probability for such items, for models whose predictions are less human-like, any divergence from human-like predictions are magnified at this end of the scale; while this would not be the case with raw probability.

Original: Furtherresearch is needed to investigate whether this or some other explanation explains these findings.
Revised: Further research is needed to investigate whether this or some other explanation explains these findings.

Original:  Conclusion
Revised:  Conclusion

Original: In this study, we investigated the mathematical relationship between the lexical predictions oflanguage models and amplitude of the human N400 response to the same word.
Revised: In this study, we investigated the mathematical relationship between the lexical predictions of language models and amplitude of the human N400 response to the same word.

Original: Specifically, wecompared whether N400 amplitude is more closely correlated with word probability or surprisal, theinformation-theoretic metric of information content.
Revised: Specifically, we compared whether N400 amplitude is more closely correlated with word probability or surprisal, the information-theoretic metric of information content.

Original: For the models whose predictions more closelymatch those of humans, surprisal was the better metric, supporting an information-theoretic accountof the neurocognitive system underlying the N400.
Revised: For the models whose predictions more closely match those of humans, surprisal was the better metric, supporting an information-theoretic account of the neurocognitive system underlying the N400.