Dependency Locality and Neural Surprisal as Predictors of Processing Difficulty: Evidence from Reading Times (2021)

Abstract: This paper compares two influential theories of processing difficulty: Gibson (2000)’s Dependency Locality Theory (DLT) and Hale (2001)’s Surprisal Theory. While prior work has aimed to compare DLT and Surprisal Theory (see Demberg and Keller, 2008), they have not yet been compared using more modern and powerful methods for estimating surprisal and DLT integration cost. I compare estimated surprisal values from two models, an RNN and a Transformer neural network, as well as DLT integration cost from a hand-parsed treebank, to reading times from the Dundee Corpus. Our results for integration cost corroborate those of Demberg and Keller (2008), finding that it is a negative predictor of reading times overall and a strong positive predictor for nouns, but contrast with their observations for surprisal, finding strong evidence for lexicalized surprisal as a predictor of reading times. Ultimately, I conclude that a broad-coverage model must integrate both theories in order to most accurately predict processing difficulty.

## introduction computational theories of language processing difficulty typically argue for either a memory or expectation-based approach @xcite . memory based models (eg. @xcite @xcite focus on the idea that resources are allocated for integrating, storing, and retrieving linguistic input. on the other hand, expectation-based models (eg. @xcite propose that resources are proportionally devoted to maintaining different potential representations, leading to an expectation-based view. @xcite @xcite . here, i focus on one representative theory from each group. the first is the dependency locality theory, or dlt, which was initially proposed by @xcite . the dlt quantifies the processing difficulty, or integration cost (ic) of discourse ref-erents (i.e. nouns and finite verbs), as the number of intervening nouns and verbs between a word and its preceding head or dependent, plus an additional cost of 1. thus, the ic is always incurred at the second word in the dependency relation in linear order. this is shown in figure 1 . note that ic only assigns a non-zero cost to discourse referents. meanwhile, @xcite and @xcite 's surprisal theory formulates the processing difficulty of a word w n in context @xmath0. . . w n-1 to be its information-theoretic surprisal, given by so that words that are more likely in context will then be assigned lower processing difficulties. some work has attempted to compare dlt and surprisal as competing predictors of processing difficulty. most notably, @xcite compared processing difficulties from dlt and surprisal to the dundee corpus @xcite , a large corpus of eye-tracking data. specifically, they examined lexicalized surprisal (where the model assigned probabilities to the words themselves), unlexicalized surprisal (where the model only had access to parts of speech), and integration cost. they found that unlexicalized surprisal was a strong predictor of reading times, while ic and lexicalized surprisal were weak predictors. they also observed that ic was a strong positive predictor of reading times for nouns, and found little correlation between ic and surprisal. notably, however, demberg and keller's study relied on older methods of calculating surprisal, using a probabilistic context free grammar (pcfg). other similar work (eg. @xcite has used n-gram models, which do not account for structural probabilities. computational language models (lms) such as n-grams and pcfgs are suboptimal for estimating the probabilities of words in context compared to humans. figure 1 : dependency locality theory integration costs however, recent work in neural network language modeling has shown that recurrent neural networks (rnns) and transformers are capable not only of learning word sequences, but also underlying syntactic structure @xcite @xcite . this makes them suited for more accurate estimations of surprisal. in this paper, i examine the correlation between reading times, dlt integration cost, and surprisal. specifically, i compare results from a manually parsed treebank for ic and two neural lms for surprisal, to eye-tracking times sourced from the dundee corpus. i additionally examine the correlation between ic and surprisal. ## methods the method in this study is similar to that of prior work on empirically testing theories of sentence processing (eg. @xcite @xcite , using reading time data in order to estimate processing difficulty. ## results there was minimal correlation between surprisal and ic across both models, and moderately high correlation between gpt-2 and rnn surprisal values ( ## discussion and conclusion this study examined the strength of two different theories of processing difficulty as predictors of eye-tracking data. overall, neural surprisal has a significant positive relationship with reading times, indicating that it is a strong candidate for a broadcoverage model of sentence processing difficulty. contrary to the predictions of dlt, there was a significant negative relationship between reading times and integration cost, as in @xcite . all data ic gpt-2 gpt-2 0.128 rnn 0.267 0.684 nouns only ic gpt-2 gpt-2 -0.0163 rnn -0.0188 0.562 this negative coefficient is likely due to the fact that dlt only makes its reading time predictions for discourse referents, assigning non-referents a processing difficulty of zero. when comparing ic solely to noun reading times, there was a strong positive coefficient, as expected. additionally, dependency locality has a well-documented crosslinguistic impact on word order @xcite @xcite , suggesting that a modified form of ic which predicts non-discourse referent processing difficulties may be a stronger and more accurate model. our results for surprisal are promising evidence that surprisal theory can accurately measure sentence processing difficulty. as hypothesised by surprisal theory, there was a positive linear effect for both gpt-2 and the rnn. this differs from @xcite , who found that lexicalized surprisal had an insignificant correlation with reading times from a grammar-based lm. as the corpus used in this study was identical to that in @xcite , these findings support work which indicates that neural lms are capable of simulating human language processing better than grammar-based lms @xcite . i also found a moderately high correlation between rnn and gpt-2 surprisal values, implying that neither model significantly differs from the other. similarly to @xcite , ic and neural surprisal were minimally correlated. when both were added as factors in a mixed effects model, the results remained similar, with ic being negative for all data, and strongly positive for nouns. given our results as a whole, this suggests that as ic is a strong predictor for nouns, a true broad-coverage model must integrate ideas from both dlt and surprisal theory. while i did not note any major gaps in predictions of surprisal, other work has found that it cannot fully account for reading time differences in ambiguities @xcite . our positive results are in part due to the fact that the dundee corpus consists mostly of common syntactic constructions, and therefore does not provide a perfect generalized picture of sentence processing. thus, this work is consistent with the hypothesis that while appealing, a broad-coverage measure of processing difficulty cannot simply use one model of processing. potential future work could aim to combine expectation-based models with memory-based theories, such that processing involves both discarding potential representations and integration into the prior structure.