Surprisal Estimators for Human Reading Times Need Character Models (2021)

Abstract: While the use of character models has been popular in NLP applications, it has not been explored much in the context of psycholinguistic modeling. This paper presents a character model that can be applied to a structural parser-based processing model to calculate word generation probabilities. Experimental results show that surprisal estimates from a structural processing model using this character model deliver substantially better fits to self-paced reading, eye-tracking, and fMRI data than those from large-scale language models trained on much more data. This may suggest that the proposed processing model provides a more humanlike account of sentence processing, which assumes a larger role of morphology, phonotactics, and orthographic complexity than was previously thought.

## introduction and related work expectation-based theories of sentence processing @xcite posit that processing difficulty is determined by predictability in context. in support of this position, predictability quantified through surprisal has been shown to correlate with behavioral measures of word processing difficulty @xcite @xcite @xcite . however, surprisal itself makes no representational assumptions about sentence processing, leaving open the question of how best to estimate its underlying probability model. in natural language processing (nlp) applications, the use of character models has been popular for several years @xcite @xcite . character models have been shown not only to alleviate problems with out-of-vocabulary words but also to embody morphological information available at the subword level. for this reason, they have been extensively used to model morphological processes @xcite or incorporate morphological information into models of syntactic acquisition @xcite . nonetheless, the use of character models has been slow to catch on in psycholinguistic surprisal estimation, which has recently focused on evaluating largescale language models that make predictions at the word level (e.g. @xcite @xcite . this raises the question of whether incorporating character-level information into an incremental processing model will result in surprisal estimates that better characterize predictability in context. to answer this question, this paper presents a character model that can be used to estimate word generation probabilities in a structural parser-based processing model. 1 the proposed model defines a process of generating a word from an underlying lemma and a morphological rule, which allows the processing model to capture the predictability of a given word form in a fine-grained manner. regression analyses on self-paced reading, eye-tracking, and fmri data demonstrate that surprisal estimates calculated from this character-based structural processing model contribute to substantially better fits compared to those calculated from large-scale language models, despite the fact that these other models are trained on much more data and show lower perplexities on test data. this finding deviates from the monotonic relationship between test perplexity and predictive power observed in previous studies @xcite . furthermore, it suggests that the character-based structural processing model may provide a more humanlike account of processing difficulty and may suggest a larger role of morphology, phonotactics, and orthographic complexity than was previously thought. ## background the experiments presented in this paper use surprisal predictors @xcite calculated by an incremental processing model based on a leftcorner parser @xcite . this incremental processing model provides a probabilistic account of sentence processing by making a single lexical attachment decision and a single grammatical attachment decision for each input word. surprisal. surprisal can be defined as the negative log ratio of prefix probabilities of word sequences w 1..t at consecutive time steps t -1 and t: these prefix probabilities can be calculated by marginalizing over the hidden states q t of the forward probabilities of an incremental processing model: these forward probabilities are in turn defined recursively using a transition model: p(w 1..t q t ) @xmath0..t-1 q t-1 ) at each time step, a left-corner parsing model generates a new word w t and a new store state q t in two phases (see figure 1 ). first, it makes a lexical decision ℓ t regarding whether to use the word to complete the most recent derivation fragment (match), or to use the word to create a new preterminal node a ℓ t (no-match). subsequently, the model makes a grammatical decision g t regarding whether to use a predicted grammar rule to combine the node constructed in the lexical phase a ℓ t with the next most recent derivation fragment (match), or to use the grammar rule to convert this node into a new derivation fragment a g t /b g t (no-match): foot_1 thus, the parser creates a hierarchically organized sequence of derivation fragments and joins these fragments up whenever expectations are satisfied. in order to update the store state based on the lexical and grammatical decisions, derivation fragments above the most recent nonterminal node are carried forward, and derivation fragments below it are set to null (⊥): ## experiment 1: effect of character model in order to assess the influence of the characterbased word generation model over the baseline word generation model on the predictive quality of surprisal estimates, linear mixed-effects models containing common baseline predictors and one or more surprisal predictors were fitted to self-paced reading times. subsequently, a series of likelihood ratio tests were conducted in order to evaluate the relative contribution of each surprisal predictor to regression model fit. ## experiment 2: comparison to other models to further examine the impact of the characterbased word generation model, charwsurp and fre-qwsurp were evaluated against surprisal predictors calculated from a number of other large-scale pretrained language models and smaller parser-based models. to compare the predictive power of surprisal estimates from different language models on equal footing, we calculated the increase in loglikelihood (∆ll) to a baseline regression model as a result of including a surprisal predictor, following recent work @xcite . ## experiment 3: eye-tracking data in order to examine whether these results generalize to other latency-based measures, linear-mixed effects models were fitted on the dundee eyetracking corpus @xcite to test the contribution of each surprisal predictor, following similar procedures to experiment 2. ## conclusion this paper presents a character model that can be used to estimate word generation probabilities in a structural parser-based processing model. experiments demonstrate that surprisal estimates calculated from this processing model generally contribute to substantially better fits to human response data than those calculated from large-scale pretrained language models or other incremental parsers. these results add a new nuance to the relationship between perplexity and predictive power reported in previous work @xcite . in addition, they suggest that structural parser-based processing models may provide a more humanlike account of sentence processing, and may suggest a larger role of morphology, phonotactics, and orthographic complexity than was previously thought. shravan vasishth. 2006. on the proper treatment of spillover in real-time reading studies: consequences for psycholinguistic theories. in proceedings of the international conference on linguistic evidence, pages 96-100. ethan gotlieb wilcox, jon gauthier, jennifer hu, peng qian, and roger p. levy. 2020. on the predictive power of neural language models for human realtime comprehension behavior. in proceedings of the 42nd annual meeting of the cognitive science society, pages 1707-1713.