Coreference-aware Surprisal Predicts Brain Response (2021)

Abstract: Recent evidence supports a role for coreference processing in guiding human expectations about upcoming words during reading, based on covariation between reading times and word surprisal estimated by a coreference-aware semantic processing model (Jaffe et al. 2020).The present study reproduces and elaborates on this finding by (1) enabling the parser to process subword information that might better approximate human morphological knowledge, and (2) extending evaluation of coreference effects from self-paced reading to human brain imaging data. Results show that an expectation-based processing effect of coreference is still evident even in the presence of the stronger psycholinguistic baseline provided by the subword model, and that the coreference effect is observed in both self-paced reading and fMRI data, providing evidence of the effect’s robustness.

## introduction coreference resolution is a core component of language that enables comprehenders to construct a detailed representation of referents throughout a discourse. extensive prior work has explored various conditions related to coreference resolution that affect language processing @xcite @xcite @xcite and have often relied on constructed stimuli to manipulate variables of interest. complementary studies using broad-coverage, naturalistic stimuli have observed coreference effects in selfpaced reading in two instantiations: (1) as a memory effect based on the count of times an entity has been previously mentioned @xcite and ( 2 ) as an expectation-based effect, operationalized by surprisal estimates from a coreferenceaware incremental parser @xcite . while expectation-based effects have been previously shown for naturalistic stimuli in self-paced reading (spr) and functional magnetic resonance imaging (fmri) @xcite , the current study extends these findings by arguing that coreference resolution contributes to predicting human behavioral data over previous implementations (e.g., surprisal) that do not model coreference. the current study elaborates on jaffe et al. (2020) by re-examining the expectation-based effect of coreference information using an improved baseline provided by an extension of the coreferenceaware incremental parser. first, the probabilities of coreference decisions are modeled using a multilayer perceptron (mlp) model, leading to improved generalizability over the previous system based on maximum-entropy. additionally, the incremental parser incorporates a character-based word generation model @xcite , which has been shown to yield surprisal estimates that predict human reading times more accurately than surprisal calculated from high-capacity neural language models. linguistic task accuracy for coreference resolution shows improvements from the extended incremental parser, further motivating its use for psycholinguistic evaluation. regression analyses are conducted using surprisal estimates from the parser to determine whether coreference awareness helps explain measures of human sentence comprehension from spr. in addition, we further evaluate whether these effects generalize to data from fmri. results from self-paced reading replicate @xcite by showing both (1) that coreference awareness improves the parser's approximation of human subjective surprisal and (2) that this improvement does not fully explain a previously reported facilitation effect from repeated mentions, which is plausibly driven by ease of memory retrieval. results from fmri support a contribution of coreferenceawareness to human surprisal estimation, but fail to support a dissociable memory effect. results from both modalities thus converge in favor of the hypothesis that human linguistic expectations are sensitive to coreferential cues, with possible additional influences of memory retrieval. a transition model captures how these distributions are related over timesteps: at a given timestep, the full generative process for the parser includes a lexical decision ℓ t , preterminal decision p t , word w t , grammatical decision g t and parser state q t : the parser also makes a coreference index decision that chooses an antecedent in a fixed window prior to the current word, or a special null index, which indicates no antecedent. this coreference decision is conditioned on the preterminal sequence up to the current timestep p 1..t , which includes syntactic category c p ℓ 1..t and predicate context h p ℓ 1..t decisions from earlier timesteps. syntactic category and predicate context are generated as part of the lexical decision ℓ t during inference, and are derived from a generalized categorial grammar reannotation @xcite of the wall street journal section of ontonotes @xcite for training. predicate contexts consist of a lemmatized predicate name and an argument number, such as pour_1, indicating the first participant in a pouring predication. together, the parser decisions generate word-by-word surprisal estimates that incorporate syntactic structure as well as propositional co-occurrences from the training data. recently, @xcite showed improved fit to self-paced reading and eye-tracking data by incorporating a character-based word generation model. their word generation model is adopted in the current work for an improved surprisal baseline for examining coreference effects. formally, the word probability from equation 3 decomposes into probabilities for the lemma x t , morphological rule r t , and word w t with the following conditioned-on variables: morphological rules that are part of the generalized categorial grammar reannotation scheme foot_0 @xcite are used to generate a list of x t , r t pairs that deterministically generate the observed word w t . the probability of the lemma x t is modeled as the probability of generating its character sequence one-by-one from a recurrent neural network (rnn) that conditions on the syntactic category and predicate context from the lexical decision, as well as the previous character. similarly, the probability of the morphological rule r t is calculated by a softmax classifier that takes as input the last hidden state of a separate rnn that receives the entire character sequence of the lemma, as well as the syntactic category and predicate context from the lexical decision. by allowing the model to posit the word's underlying structure, the parser is better able to handle out-of-vocabulary words. ## methods the current work attempts to replicate jaffe et al. cdr models individual predictor response functions and convolves them to generate a continuous prediction of blood oxygenation level-dependent (bold) signals as the combination of previous events. since the effect of a predictor on the response variable is modeled as an impulse function, predictors can have varying amplitude and decay over time. this approach therefore allows predictor and response variables to have different temporal granularity. for model details, see appendix a. for each fmri experiment, two models are fit which differ minimally by the addition of a fixed effect for the predictor of interest (all models include by-subject random effects for all predictors), and correlation coefficients are calculated between each model's predictions and the fmri observations. the difference between correlation coeffi-cients across models provides the test statistic that is probed for significance by running a permutation test, where 10,000 permuted runs are generated to find the likelihood of the differences being at least as extreme as the observed difference. ## conclusion this study reproduces previously reported coreference effects in self-paced reading using an improved surprisal estimator baseline, finding evidence for a coreference-driven expectation effect during naturalistic reading. additionally, a new analysis using fmri data shows that coreferenceaware surprisal contributes to significantly better fit, further supporting the overall claim that expectation-based language processing utilizes coreferential cues. however, a memory retrieval effect for coreference is observed in spr but not in fmri, highlighting the complex nature of human coreference processing and offering potential future directions for investigation.