Original: The more human-like the language model, the more surprisal is the best predictor of N400 amplitude
Revised: The more human-like the language model, the more surprisal is the best predictor of N400 amplitude

Original: Abstract
Revised: Abstract

Original: Under information-theoretic accounts of language comprehension, the effort required to process a word is correlated with surprisal, the negative log-probability of that word given its context.
Revised: Under information-theoretic accounts of language comprehension, the effort required to process a word is correlated with surprisal, the negative log-probability of that word given its context.

Original: This can (equivalently) be considered to reflect cognitive effort in proportion to the amount of information conveyed by a given word (Frank et al., 2015), or the amount of effort required to update the our incremental predictions about upcoming words (Levy, 2008; Aurnhammer and Frank, 2019).
Revised: This can (equivalently) be considered to reflect cognitive effort in proportion to the amount of information conveyed by a given word (Frank et al., 2015), or the amount of effort required to update the our incremental predictions about upcoming words (Levy, 2008; Aurnhammer and Frank, 2019).

Original: In contrast, others (e.g. Brothers and Kuperberg, 2021) have argued that processing difficulty is proportional to the contextual probability of a word, thus positing a linear (rather than logarithmic) relationship between word probability and processing difficulty.
Revised: In contrast, others (e.g. Brothers and Kuperberg, 2021) have argued that processing difficulty is proportional to the contextual probability of a word, thus positing a linear (rather than logarithmic) relationship between word probability and processing difficulty.

Original: We investigate which of these two accounts best explain the N400, a neural response that provides some of the best evidence for prediction in language comprehension (Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020).
Revised: We investigate which of these two accounts best explain the N400, a neural response that provides some of the best evidence for prediction in language comprehension (Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020).

Original: To do this, we expand upon previous work by comparing how well the probability and surprisal calculated by 43 transformer language models predict N400 amplitude.
Revised: To do this, we expand upon previous work by comparing how well the probability and surprisal calculated by 43 transformer language models predict N400 amplitude.

Original: We thus investigate both which models’ predictions best predict the N400, and for each model, whether surprisal or probability is more closely correlated with N400 amplitude.
Revised: We thus investigate both which models’ predictions best predict the N400, and for each model, whether surprisal or probability is more closely correlated with N400 amplitude.

Original: We find that of the models tested, OPT6.7B and GPT-J are reliably the best at predicting N400 amplitude, and that for these transformers, surprisal is the better predictor.
Revised: We find that of the models tested, OPT6.7B and GPT-J are reliably the best at predicting N400 amplitude, and that for these transformers, surprisal is the better predictor.

Original: In fact, we find that the more highly correlated the predictions of a language model are with N400 amplitude, the greater the extent to which surprisal is a better predictor than probability.
Revised: In fact, we find that the more highly correlated the predictions of a language model are with N400 amplitude, the greater the extent to which surprisal is a better predictor than probability.

Original: Since language models that more closely mirror human statistical knowledge are more likely to be informative about the human predictive system, these results support the information-theoretic account of language comprehension.
Revised: Since language models that more closely mirror human statistical knowledge are more likely to be informative about the human predictive system, these results support the information-theoretic account of language comprehension.

Original:  Introduction
Revised:  Introduction

Original: If the language comprehension system follows information-theoretic principles, we should expect that the information content conveyed by a word should play a crucial role in how that word is processed.
Revised: If the language comprehension system follows information-theoretic principles, we should expect that the information content conveyed by a word should play a crucial role in how that word is processed.

Original: This is a core feature of surprisal theory : the idea that the difficulty in processing a word in context is correlated with the surprisal of that word, that is, the negative logarithm of the probability of the word in context (Hale, 2001, 2003; Levy, 2008; Smith and Levy, 2011, 2013; Frank et al., 2015; Aurnhammer and Frank, 2019; Michaelov and Bergen, 2020).
Revised: This is a core feature of surprisal theory : the idea that the difficulty in processing a word in context is correlated with the surprisal of that word, that is, the negative logarithm of the probability of the word in context (Hale, 2001, 2003; Levy, 2008; Smith and Levy, 2011, 2013; Frank et al., 2015; Aurnhammer and Frank, 2019; Michaelov and Bergen, 2020).

Original: A key mathematical property of lexical surprisal is that it reflects the information content of a word (as formally defined information-theoretically), but also, as noted by Levy (2008), is equivalent to the Kullback-Leibler divergence (Kullback and Leibler, 1951) between the predicted probability distribution for the next word and the true probability distribution (1 for the actual word and 0 for all other possible words).
Revised: A key mathematical property of lexical surprisal is that it reflects the information content of a word (as formally defined information-theoretically), but also, as noted by Levy (2008), is equivalent to the Kullback-Leibler divergence (Kullback and Leibler, 1951) between the predicted probability distribution for the next word and the true probability distribution (1 for the actual word and 0 for all other possible words).

Original: Thus, surprisal as a metric can be thought to account not only for the increased effort required to process a word with high information content (Hale, 2001; Frank et al., 2015), but also the effort required to update the entire predicted probability distribution based on encountering a word.
Revised: Thus, surprisal as a metric can be thought to account not only for the increased effort required to process a word with high information content (Hale, 2001; Frank et al., 2015), but also the effort required to update the entire predicted probability distribution based on encountering a word.

Original: An alternative to surprisal theory is what Brothers and Kuperberg (2021) term the proportional preactivation account , which also posits that prediction plays a key role in language comprehension.
Revised: An alternative to surprisal theory is what Brothers and Kuperberg (2021) term the proportional preactivation account , which also posits that prediction plays a key role in language comprehension.

Original: However, instead of processing difficulty reflecting the surprisal of a word in context, it reflects the extent to which the word was not predicted—that is, instead of − log ( p ), scales with 1 − p .
Revised: However, instead of processing difficulty reflecting the surprisal of a word in context, it reflects the extent to which the word was not predicted—that is, 1 − p rather than − log ( p ) .

Original: Thus, under surprisal theory, the mathematical relationship between lexical probability and processing difficulty is logarithmic, while under the proportional preactivation account, it is linear.
Revised: Thus, under surprisal theory, the mathematical relationship between lexical probability and processing difficulty is logarithmic, while under the proportional preactivation account, it is linear.

Original: To measure real-time human language processing, we turn to the N400, a neural response that offers a snapshot into the neurocognitive system underlying language comprehension.
Revised: To measure real-time human language processing, we turn to the N400, a neural response that offers a snapshot into the neurocognitive system underlying language comprehension.

Original: It is widely agreed that the N400 reflects the brain activation driven by encountering a word and that it is modulated by the extent to which the word (or its semantic features) are preactivated by the preceding context (Kutas and Federmeier, 2011; Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020;
Revised: It is widely agreed that the N400 reflects the brain activation driven by encountering a word and that it is modulated by the extent to which the word (or its semantic features) are preactivated by the preceding context (Kutas and Federmeier, 2011; Kutas et al., 2011; Van Petten and Luka, 2012; Kuperberg et al., 2020;

Original: Federmeier, 2021).
Revised: Federmeier, 2021).

Original: Thus, the N400 is perhaps the most suitable processing difficulty metric for evaluating the predictions of surprisal theory (Michaelov and Bergen, 2020).
Revised: Thus, the N400 is perhaps the most suitable processing difficulty metric for evaluating the predictions of surprisal theory (Michaelov and Bergen, 2020).

Original: Despite this fact, to the best of our knowledge, the question of whether surprisal or raw probability is a better predictor of N400 amplitude has only been tested for two language models thus far—Yan and Jaeger (2020) use a mixture of a 5-gram model and ‘skip bi-gram’ (both trained by Frank and Willems, 2017), and Szewczyk and Federmeier (2022) use GPT-2.
Revised: Despite this fact, to the best of our knowledge, the question of whether surprisal or raw probability is a better predictor of N400 amplitude has only been tested for two language models thus far—Yan and Jaeger (2020) use a mixture of a 5-gram model and ‘skip bi-gram’ (both trained by Frank and Willems, 2017), and Szewczyk and Federmeier (2022) use GPT-2.

Original: Both find that surprisal better fits the N400 data, though on one dataset, Szewczyk and Federmeier (2022) find that when restricting their analysis to only expected (higher-probability) items, raw probability appears to be a better predictor; while when restricting their analysis to unexpected (lower-probability items), surprisal is the better predictor.
Revised: Both find that surprisal better fits the N400 data, though on one dataset, Szewczyk and Federmeier (2022) find that when restricting their analysis to only expected (higher-probability) items, raw probability appears to be a better predictor; while when restricting their analysis to unexpected (lower-probability items), surprisal is the better predictor.

Original: In the present study, we substantially expand upon this previous work.
Revised: In the present study, we substantially expand upon this previous work.

Original: We run the stimuli from a previously-published N400 study (Nieuwland et al., 2018) through 43 neural language models, calculate the probability and surprisal for each item with each model, and use these to predict N amplitude.
Revised: We run the stimuli from a previously-published N400 study (Nieuwland et al., 2018) through 43 neural language models, calculate the probability and surprisal for each item with each model, and use these to predict N amplitude.

Original: First, we ask which model is the best predictor of N400 amplitude, and whether surprisal or raw probability is a better predictor in all models.
Revised: First, we ask which model is the best predictor of N400 amplitude, and whether surprisal or raw probability is a better predictor in all models.

Original: Additionally, because models that behave more similarly to humans overall are also more likely to be more informative about the human predictive system, we also compare surprisal and probability across models as a function of how well they predict the human N400 response overall.
Revised: Additionally, because models that behave more similarly to humans overall are also more likely to be more informative about the human predictive system, we also compare surprisal and probability across models as a function of how well they predict the human N400 response overall.

Original:  Method
Revised:  Method

Original: Following research showing that N400 amplitude is best predicted by transformers (Merkx and Frank, 2021; Michaelov et al., 2022), we restricted our analysis to contemporary transformer language models made available through the transformers (Wolf et al., 2020)
Revised: Following research showing that N400 amplitude is best predicted by transformers (Merkx and Frank, 2021; Michaelov et al., 2022), we restricted our analysis to contemporary transformer language models made available through the transformers (Wolf et al., 2020)

Original: Python package.
Revised: Python package.

Original: These included 43 models of the GPT-2 (Radford et al., 2019),
Revised: These included 43 models of the GPT-2 (Radford et al., 2019),

Original: GPT-Neo (Black et al., 2021), GPT-J (Wang and
Revised: GPT-Neo (Black et al., 2021), GPT-J (Wang and

Original: Komatsuzaki, 2021), OPT (Zhang et al., 2022), (Devlin et al., 2019), RoBERTa (Liu et al., 2019),
Revised: Komatsuzaki, 2021), OPT (Zhang et al., 2022), (Devlin et al., 2019), RoBERTa (Liu et al., 2019),

Original: ALBERT (Lan et al., 2020), XGLM (Lin et al., 2021), BLOOM (BigScience, 2022), Multilingual
Revised: ALBERT (Lan et al., 2020), XGLM (Lin et al., 2021), BLOOM (BigScience, 2022), Multilingual

Original: BERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), and XLM-R (Conneau et al., 2020;
Revised: BERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), and XLM-R (Conneau et al., 2020;

Original: Goyal et al., 2021) architectures.
Revised: Goyal et al., 2021) architectures.

Original: All models used are shown in Figure 1.
Revised: All models used are shown in Figure 1.

Original: For our N400 data, we used a subset of the data from an experiment carried out by (Nieuwland et al., 2018), a large-scale study with 334 participants.
Revised: For our N400 data, we used a subset of the data from an experiment carried out by (Nieuwland et al., 2018), a large-scale study with 334 participants.

Original: Specifically, we look at the N400 amplitudes elicited by nouns that are either more or less contextually predictable.
Revised: Specifically, we look at the N400 amplitudes elicited by nouns that are either more or less contextually predictable.

Original: Nieuwland et al.
Revised: Nieuwland et al.

Original: (2018) operationalize
Revised: (2018) operationalize

Original: N400 amplitude as mean voltage between 200-500ms after stimulus presentation at 6 electrodes in their region of interest (Cz, C3, C4, Pz, P3, and P4).
Revised: N400 amplitude as mean voltage between 200-500ms after stimulus presentation at 6 electrodes in their region of interest (Cz, C3, C4, Pz, P3, and P4).

Original: To investigate how well the language models’ predictions correlate with N400 amplitude, we ran each of the 160 stimulus sentences from the Nieuwland et al.
Revised: To investigate how well the language models’ predictions correlate with N400 amplitude, we ran each of the 160 stimulus sentences from the Nieuwland et al.

Original: (2018) study up until the critical noun through each of the the 43 language models, calculating the probability and surprisal of each of the critical nouns.
Revised: (2018) study up until the critical noun through each of the the 43 language models, calculating the probability and surprisal of each of the critical nouns.

Original: Not all critical nouns were in each model’s vocabulary.
Revised: Not all critical nouns were in each model’s vocabulary.

Original: The fairest approach is to compare only words appearing in the vocabulary of all models (see Michaelov and Bergen, 2020;
Revised: The fairest approach is to compare only words appearing in the vocabulary of all models (see Michaelov and Bergen, 2020;

Original: Michaelov et al., 2022 for similar work following this approach).
Revised: Michaelov et al., 2022 for similar work following this approach).

Original: However, with stimuli in this study, this means that only data from 37 sentences could be analyzed.
Revised: However, with stimuli in this study, this means that only data from 37 sentences could be analyzed.

Original: An alternative approach is to predict each sub-word token of the critical noun given the preceding context and the preceding sub-word tokens, and then either taking the product of probabilities or sum of the surprisals to get an overall
Revised: An alternative approach is to predict each sub-word token of the critical noun given the preceding context and the preceding sub-word tokens, and then either take the product of probabilities or sum of the surprisals to get an overall

Original: b i g
Revised: b i g

Original: sc i en c e − b l oo m − 7b b
Revised: sc i en c e − b l oo m − 7b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 3b x
Revised: sc i en c e − b l oo m − 3b x

Original: l m − m l m − 17 −b
Revised: l m − m l m − 17 −b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b b
Revised: sc i en c e − b l oo m − 1b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b a l be r t − ba s e − vbe r t − l a r ge − un c a s ed
Revised: sc i en c e − b l oo m − 1b a l be r t − ba s e − vbe r t − l a r ge − un c a s ed

Original: G eo t r end − be r t − ba s e − 15 l ang − c a s ed be r t − ba s e − m u l t ili ngua l − c a s ed
Revised: G eo t r end − be r t − ba s e − 15 l ang − c a s ed be r t − ba s e − m u l t ili ngua l − c a s ed

Original: G eo t r end − be r t − ba s e − 10 l ang − c a s ed be r t − ba s e − c a s ed be r t − ba s e − un c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e b
Revised: G eo t r end − be r t − ba s e − 10 l ang − c a s ed be r t − ba s e − c a s ed be r t − ba s e − un c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 560 m be r t − l a r ge − c a s ed f a c eboo k − op t −m x l m − m l m − 100 − f a c eboo k − op t −m f a c eboo
Revised: sc i en c e − b l oo m − 560 m be r t − l a r ge − c a s ed f a c eboo k − op t −m x l m − m l m − 100 − f a c eboo k − op t −m f a c eboo

Original: k − x g l m − 564 M gp tgp t 2 − m ed i u m
Revised: k − x g l m − 564 M gp tgp t 2 − m ed i u m

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo − 125 M x l m − r obe r t a − l a r ge
Revised: I − gp t − neo − 125 M x l m − r obe r t a − l a r ge

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo −r obe r t a − ba s e f a c eboo k − x g l m −be r t − l
Revised: I − gp t − neo −r obe r t a − ba s e f a c eboo k − x g l m −be r t − l

Original: a r ge − un c a s ed − w ho l e − w o r d − m a sk i ng be r t − l a r ge − c a s ed − w ho l e − w o r d − m a
Revised: a r ge − un c a s ed − w ho l e − w o r d − m a sk i ng be r t − l a r ge − c a s ed − w ho l e − w o r d − m a

Original: sk
Revised: sk

Original: i ng f a c eboo k − op t
Revised: i ng f a c eboo k − op t

Original: −f a c eboo k − x g l m −f a c eboo k − x g l m −f a c eboo k − op t −
Revised: −f a c eboo k − x g l m −f a c eboo k − x g l m −f a c eboo k − op t −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo − 2 gp t 2 − x l gp t 2 − l a r ge a l be r t − x l a r ge − vf a c eboo k − x g l m −a l be r t − xx l a r ge − vr obe r t a − l a r ge f a c eboo k − op t −
Revised: I − gp t − neo − 2 gp t 2 − x l gp t 2 − l a r ge a l be r t − x l a r ge − vf a c eboo k − x g l m −a l be r t − xx l a r ge − vr obe r t a − l a r ge f a c eboo k − op t −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − j − 6 B f a c eboo k − x l m − r obe r t a − x l
Revised: I − gp t − j − 6 B f a c eboo k − x l m − r obe r t a − x l

Original: Model
Revised: Model

Original: x l m − m l m − 17 −be r t − ba s e − c a s ed be r t − l a r ge − c a s ed be r t − ba s e − un c a s ed be r t − l
Revised: x l m − m l m − 17 −be r t − ba s e − c a s ed be r t − l a r ge − c a s ed be r t − ba s e − un c a s ed be r t − l

Original: a r ge − un c a s ed b
Revised: a r ge − un c a s ed b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b b
Revised: sc i en c e − b l oo m − 1b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 7b x
Revised: sc i en c e − b l oo m − 7b x

Original: l m − m l m − 100 − b
Revised: l m − m l m − 100 − b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 1b b
Revised: sc i en c e − b l oo m − 1b b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 560 m
Revised: sc i en c e − b l oo m − 560 m

Original: b
Revised: b

Original: i g
Revised: i g

Original: sc i en c e − b l oo m − 3b be r t − ba s e − m u l t ili ngua l − c a s ed
Revised: sc i en c e − b l oo m − 3b be r t − ba s e − m u l t ili ngua l − c a s ed

Original: G eo t r end − be r t − ba s e − 15 l ang − c a s ed a l be r t − ba s e − v
Revised: G eo t r end − be r t − ba s e − 15 l ang − c a s ed a l be r t − ba s e − v

Original: G eo t r end − be r t − ba s e − 10 l ang − c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e be r t − l a r ge − c a s ed − w ho l e − w o r d − m a
Revised: G eo t r end − be r t − ba s e − 10 l ang − c a s ed a l be r t − l a r ge − vx l m − r obe r t a − ba s e be r t − l a r ge − c a s ed − w ho l e − w o r d − m a

Original: sk i ng a l be r t − x l a r ge − vx l m − r obe r t a − l a r ge f a c eboo k − x l m − r obe r t a − x l be r t − l a r ge − un c a s ed − w ho l e − w o r d − m a
Revised: sk i ng a l be r t − x l a r ge − vx l m − r obe r t a − l a r ge f a c eboo k − x l m − r obe r t a − x l be r t − l a r ge − un c a s ed − w ho l e − w o r d − m a

Original: sk i ng a l be r t − xx l a r ge − vf a c eboo
Revised: sk i ng a l be r t − xx l a r ge − vf a c eboo

Original: k − x g l m − 564 M r obe r t a − ba s e
Revised: k − x g l m − 564 M r obe r t a − ba s e

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo − 125 M gp tf a c eboo k − op t −m f a c eboo k − op t −m r obe r t a − l a r ge f a c eboo k − x g l m −f a c eboo k − x g l m −gp t 2 − m ed i u m gp t 2 − x l gp t 2 − l a r ge f a c eboo k − x g l m −
Revised: I − gp t − neo − 125 M gp tf a c eboo k − op t −m f a c eboo k − op t −m r obe r t a − l a r ge f a c eboo k − x g l m −f a c eboo k − x g l m −gp t 2 − m ed i u m gp t 2 − x l gp t 2 − l a r ge f a c eboo k − x g l m −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo −f a c eboo k − x g l m −
Revised: I − gp t − neo −f a c eboo k − x g l m −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − neo −f a c eboo k − op t −f a c eboo k − op t −f a c eboo k − op t −
Revised: I − gp t − neo −f a c eboo k − op t −f a c eboo k − op t −f a c eboo k − op t −

Original: E l eu t he r A
Revised: E l eu t he r A

Original: I − gp t − j − 6 B
Revised: I − gp t − j − 6 B

Original: Model
Revised: Model

Original: All
Revised: All

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Model
Revised: Model

Original: [Table]
Revised: [Table]

Original: Probability
Revised: Probability

Original: Surprisal word probability or surprisal.
Revised: Surprisal word probability or surprisal.

Original: This produces a probability and surprisal for each model; however, given the differences in tokenization (including tokenizer algorithm) and training tasks (for example, traditional vs. masked language modeling, and the possible addition of whole-word masking to the latter task), this may introduce possible confounds.
Revised: This produces a probability and surprisal for each model; however, given the differences in tokenization (including tokenizer algorithm) and training tasks (for example, traditional vs. masked language modeling, and the possible addition of whole-word masking to the latter task), this may introduce possible confounds.

Original: Thus, in our analyses, we try both approaches.
Revised: Thus, in our analyses, we try both approaches.

Original: Probability, surprisal, and N400 amplitude were all z-scored.
Revised: Probability, surprisal, and N400 amplitude were all z -scored.

Original:  Results 3.1 Single-token critical nouns
Revised:  Results 3.1 Single-token critical nouns

Original: First, we run our analyses for only the critical nouns that are single tokens in all models.
Revised: First, we run our analyses for only the critical nouns that are single tokens in all models.

Original: We constructed linear mixed-effects regressions following (Michaelov et al., 2022), with the relevant language model probability or surprisal as a main effect, with the laboratory that the experiment was carried out in as a covariate, and random intercepts of experimental subject and stimulus item.
Revised: We constructed linear mixed-effects regressions following (Michaelov et al., 2022), with the relevant language model probability or surprisal as a main effect, with the laboratory that the experiment was carried out in as a covariate, and random intercepts of experimental subject and stimulus item.

Original: We then calculated each regression’s Akaike Information Criterion (AIC; Akaike, 1973) as a metric of fit.
Revised: We then calculated each regression’s Akaike Information Criterion (AIC; Akaike, 1973) as a metric of fit.

Original: We then compared regression fit across model and metric.
Revised: We then compared regression fit across model and metric.

Original: The results can be seen in Figure 1A.
Revised: The results can be seen in Figure 1A.

Original: First, in line with previous work, for the models that can be used to best predict N400 amplitude—
Revised: First, in line with previous work, for the models that can be used to best predict N400 amplitude—

Original: XLM-R XL , GPT-J, OPT-6B,
Revised: XLM-R XL , GPT-J, OPT-6B,

Original: etc.—surprisal better predicts N400 amplitude than raw probability.
Revised: etc.—surprisal better predicts N400 amplitude than raw probability.

Original: However, in contrast to previous work, probability is a better predictor for some other models.
Revised: However, in contrast to previous work, probability is a better predictor for some other models.

Original: To test for trends across all models, we ran a likelihood ratio test using linear mixed-effects regressions.
Revised: To test for trends across all models, we ran a likelihood ratio test using linear mixed-effects regressions.

Original: AIC was the dependent variable, and we compared a null model with only language model as random intercept with one also including metric (either probability or surprisal) as a fixed effect, finding no significant difference ( χ 2 (1) = 3 . 62 , p = 0 . 0572 ).
Revised: AIC was the dependent variable, and we compared a null model with only language model as random intercept with one also including metric (either probability or surprisal) as a fixed effect, finding no significant difference ( χ 2 (1) = 3 . 62 , p = 0 . 0572 ).

Original: Thus, there is no overall trend as to whether surprisal or probability is a better predictor of N400 amplitude.
Revised: Thus, there is no overall trend as to whether surprisal or probability is a better predictor of N400 amplitude.

Original: However, as can be seen in the graph, it appears that for models whose predictions are better predictors of N400 amplitude, surprisal tends to be a better predictor of N400 amplitude than probability.
Revised: However, as can be seen in the graph, it appears that for models whose predictions are better predictors of N400 amplitude, surprisal tends to be a better predictor of N400 amplitude than probability.

Original: In other words, it appears that as model performance at N400 amplitude prediction improves, the extent to which surprisal is a better predictor than probability increases.
Revised: In other words, it appears that as model performance at N400 amplitude prediction improves, the extent to which surprisal is a better predictor than probability increases.

Original: To test this, we constructed a simple least-squares regression with the surprisal regression AICs as the independent variable and the difference between surprisal and probability regression AICs as the dependent variable.
Revised: To test this, we constructed a simple least-squares regression with the surprisal regression AICs as the independent variable and the difference between surprisal and probability regression AICs as the dependent variable.

Original: We found a significant correlation between the two ( F (1 , 41)
Revised: We found a significant correlation between the two ( F (1 , 41)

Original: = 11 .
Revised: = 11 .

Original: 06 , p = 0 . 0019 )
Revised: 06 , p = 0 . 0019 )

Original: in the previously-discussed direction—as surprisal regression AIC decreases, the extent to which surprisal regression AIC is lower than probability regression AIC increases.
Revised: in the previously-discussed direction—as surprisal regression AIC decreases, the extent to which surprisal regression AIC is lower than probability regression AIC increases.

Original: We also constructed a least-squares regression in the same vein but with probability regression AIC as the independent variable; which also displayed a significant correlation ( F (1 , 41) = 90 .
Revised: We also constructed a least-squares regression in the same vein but with probability regression AIC as the independent variable; which also displayed a significant correlation ( F (1 , 41) = 90 .

Original: 15 , p < 0 . 0001 ).
Revised: 15 , p < 0 . 0001 ).

Original: Thus, we find that as language model performance at N400 prediction (whether based on surprisal or raw probability) increases, so does the extent to which surprisal is a better predictor.
Revised: Thus, we find that as language model performance at N400 prediction (whether based on surprisal or raw probability) increases, so does the extent to which surprisal is a better predictor.

Original: 3.2 All critical words
Revised: 3.2 All critical words

Original: We then ran the same analyses on all the stimuli, where the critical nouns were either one or more tokens in all models.
Revised: We then ran the same analyses on all the stimuli, where the critical nouns were either one or more tokens in all models.

Original: As before, we constructed linear mixed-effects regressions in the same way, using AIC as a metric of fit.
Revised: As before, we constructed linear mixed-effects regressions in the same way, using AIC as a metric of fit.

Original: The results are shown inFigure 1B.
Revised: The results are shown inFigure 1B.

Original: On this analysis, the ranking of which models best predict N400 amplitude differs.
Revised: On this analysis, the ranking of which models best predict N400 amplitude differs.

Original: But the higher-level patterns do not: there is no significant difference in performance between surprisal and probability overall ( χ 2 (1) = 0 .
Revised: But the higher-level patterns do not: there is no significant difference in performance between surprisal and probability overall ( χ 2 (1) = 0 .

Original: 97 , p = 0 . 3258 ), and  regression AIC ( F (1 , 41)
Revised: 97 , p = 0 . 3258 ), and surprisal regression AIC ( F (1 , 41)

Original: = 218 , p < 0 . 0001 ) and probability regression AIC ( F (1 , 41) = 30 .
Revised: = 218 , p < 0 . 0001 ) and probability regression AIC ( F (1 , 41) = 30 .

Original: 6 , p < 0 . 0001 ) are correlated with the difference between the two in the same direction—that is, as the extent to which the predictions of a language model predict N400 amplitude improves, so does the extent to which surprisal is a better predictor.
Revised: 6 , p < 0 . 0001 ) are correlated with the difference between the two in the same direction—that is, as the extent to which the predictions of a language model predict N400 amplitude improves, so does the extent to which surprisal is a better predictor.

Original:  Discussion
Revised:  Discussion

Original: Clear patterns are seen in both sets of results.
Revised: Clear patterns are seen in both sets of results.

Original: Under both analyses, neither surprisal nor probability is better overall, but for the models that best predict N400 amplitude (when either surprisal or probability is considered), surprisal is the best predictor.
Revised: Under both analyses, neither surprisal nor probability is better overall, but for the models that best predict N400 amplitude (when either surprisal or probability is considered), surprisal is the best predictor.

Original: This in fact is shown to be a statistically significant regularity: and as the performance of language model predictions at predicting N400 amplitude improves, the extent to which surprisal is a better predictor than probability increases.
Revised: This in fact is shown to be a statistically significant regularity: and as the performance of language model predictions at predicting N400 amplitude improves, the extent to which surprisal is a better predictor than probability increases.

Original: The question of which model best predicts N400 amplitude is more complicated.
Revised: The question of which model best predicts N400 amplitude is more complicated.

Original: Masked language models appear to perform worse at modeling N400 amplitude when all tokens are considered compared to the when only single-token words are considered.
Revised: Masked language models appear to perform worse at modeling N400 amplitude when all tokens are considered compared to the when only single-token words are considered.

Original: This is striking given that under the single-token-only analysis, the best-performing model (XLM-R XL ) is a masked language model.
Revised: This is striking given that under the single-token-only analysis, the best-performing model (XLM-R XL ) is a masked language model.

Original: This pattern is likely at least partly explained by training task.
Revised: This pattern is likely at least partly explained by training task.

Original: Specifically, predicting later sub-word tokens based only on their preceding context and preceding sub-word tokens occurs with every single multi-token word for autoregressive language models, but for masked language models, this is not necessarily the case.
Revised: Specifically, predicting later sub-word tokens based only on their preceding context and preceding sub-word tokens occurs with every single multi-token word for autoregressive language models, but for masked language models, this is not necessarily the case.

Original: On the one hand, this suggests that if attempting to test which model best predicts a human comprehension metric like the N400, it may be best to use single-token-only analyses.
Revised: On the one hand, this suggests that if attempting to test which model best predicts a human comprehension metric like the N400, it may be best to use single-token-only analyses.

Original: On the other hand, however, from a cognitive modeling perspective, it is valuable to be able to analyze all stimuli, especially if the surprisal of a sequence of two or more words is needed.
Revised: On the other hand, however, from a cognitive modeling perspective, it is valuable to be able to analyze all stimuli, especially if the surprisal of a sequence of two or more words is needed.

Original: Our results suggest that for such experiments, autoregressive models such as GPT-J and OPT-6B (the best-performing in our experiment) should be used.
Revised: Our results suggest that for such experiments, autoregressive models such as GPT-J and OPT-6B (the best-performing in our experiment) should be used.

Original: We now return to the question of how well an information-theoretic account of language processing, surprisal theory, accounts for the experimental neurocognitive data, compared to the proportional preactivation account.
Revised: We now return to the question of how well an information-theoretic account of language processing, surprisal theory, accounts for the experimental neurocognitive data, compared to the proportional preactivation account.

Original: The results show that for models with more human-like predictions, surprisal better fits the data.
Revised: The results show that for models with more human-like predictions, surprisal better fits the data.

Original: Because these models are more likely to also reveal the mathematical relationship between lexical statistics and human predictive processing, this result supports the informationtheoretic account.
Revised: Because these models are more likely to also reveal the mathematical relationship between lexical statistics and human predictive processing, this result supports the informationtheoretic account.

Original: Finally, we may want to ask why it is the case that for less human-like models, probability better models N400 amplitude.
Revised: Finally, we may want to ask why it is the case that for less human-like models, probability better models N400 amplitude.

Original: One possible explanation relates to Szewczyk and Federmeier’s (2022) finding that language model probability is more closely correlated with N400 amplitude for higherprobability items, while surprisal is more closely correlated for lower-probability items.
Revised: One possible explanation relates to Szewczyk and Federmeier’s (2022) finding that language model probability is more closely correlated with N400 amplitude for higherprobability items, while surprisal is more closely correlated for lower-probability items.

Original: Because surprisal is a logarithmic metric, small differences at the extremely low-probability end of the scale are magnified.
Revised: Because surprisal is a logarithmic metric, small differences at the extremely low-probability end of the scale are magnified.

Original: Thus, while there may only be small differences in probability for such items, for models whose predictions are less human-like, any divergence from human-like predictions are magnified at this end of the scale; while this would not be the case with raw probability.
Revised: Thus, while there may only be small differences in probability for such items, for models whose predictions are less human-like, any divergence from human-like predictions are magnified at this end of the scale; while this would not be the case with raw probability.

Original: Further research is needed to investigate whether this or some other explanation explains these findings.
Revised: Further research is needed to investigate whether this or some other explanation explains these findings.

Original:  Conclusion
Revised:  Conclusion

Original: In this study, we investigated the mathematical relationship between the lexical predictions of language models and amplitude of the human N400 response to the same word.
Revised: In this study, we investigated the mathematical relationship between the lexical predictions of language models and amplitude of the human N400 response to the same word.

Original: Specifically, we compared whether N400 amplitude is more closely correlated with word probability or surprisal, the information-theoretic metric of information content.
Revised: Specifically, we compared whether N400 amplitude is more closely correlated with word probability or surprisal, the information-theoretic metric of information content.

Original: For the models whose predictions more closely match those of humans, surprisal was the better metric, supporting an information-theoretic account of the neurocognitivesystem underlying the N400.
Revised: For the models whose predictions more closely match those of humans, surprisal was the better metric, supporting an information-theoretic account of the neurocognitive