Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities (2022)

Abstract: Humans exhibit garden path effects: When reading sentences that are temporarily structurally ambiguous, they slow down when the structure is disambiguated in favor of the less preferred alternative. Surprisal theory (Hale, 2001; Levy, 2008), a prominent explanation of this finding, proposes that these slowdowns are due to the unpredictability of each of the words that occur in these sentences. Challenging this hypothesis, van Schijndel and Linzen (2021) find that estimates of the cost of word predictability derived from language models severely underestimate the magnitude of human garden path effects. In this work, we consider whether this underestimation is due to the fact that humans weight syntactic factors in their predictions more highly than language models do. We propose a method for estimating syntactic predictability from a language model, allowing us to weigh the cost of lexical and syntactic predictability independently. We find that treating syntactic predictability independently from lexical predictability indeed results in larger estimates of garden path. At the same time, even when syntactic predictability is independently weighted, surprisal still greatly underestimate the magnitude of human garden path effects. Our results support the hypothesis that predictability is not the only factor responsible for the processing cost associated with garden path sentences.

## introduction readers exhibit garden path effects: when reading a temporarily syntactically ambiguous sentence, they tend to slow down when the sentence is disambiguated in favor of the less preferred parse. for example, a participant who reads the sentence fragment (1) the suspect sent the file . . . can construct a partial parse in at least two distinct ways: in one reading, the verb sent acts as the main verb of the sentence, and the continuation of the sentence as an additional argument to sent (as in 1a). in another, less likely, reading, sent the file acts as a modifier in a complex subject, which then requires an additional verb phrase to form a complete sentence (as in 1b). prior work has demonstrated that regions like deserved further investigation, which disambiguate these temporarily ambiguous sentences in favor of the modifier parse (1b), are read slower than those same words would be in an unambiguous version of sentence, such as the following: in (2), the presence of who was signals to the reader that sent the file acts as a modifier @xcite . one account of this phenomenon, surprisal theory @xcite , suggests that readers maintain a probabilistic representation of all possible parses of the input as they process the sentence incrementally. processing difficulty in garden path sentences is the cost associated with updating this representation; this cost is proportional to the negative log probability, or surprisal, of the newly observed material under the reader's model of upcoming words. this theory predicts that the slowdown associated with garden path sentences can be entirely captured by the differences in surprisal between the disambiguating region in ambiguous garden path sentences and that same region in a matched unambiguous sentence. van @xcite tested this hypothesis. they estimated the surprisals associated with garden path sentences using lstm language models (lms) trained over large natural language corpora. based on the core assumption of surprisal theory-that processing difficulty on a word, when all lexical factors are kept constant, stands in a constant proportion to the word's surprisal, regardless of its syntactic context-they estimated a conversion factor between surprisal and reading times from non-garden path sentences. applying this conversion factor to the critical words in garden path sentences, van schijndel and linzen found that surprisal theory, when paired with the surprisals estimated by their models, severely underestimated the magnitude of the garden path effect for three garden path constructions, consistent with attempts to estimate the magnitude of other syntacticallymodulated effects @xcite . moreover, the predicted reading times did not correctly predict differences across the difference garden path constructions, suggesting that no single conversion factor between surprisal and reading times could predict the magnitude of the garden path effect in all three constructions. the underestimation documented by van schijndel and linzen can be interpreted in one of two ways: either (1) surprisal theory cannot, on its own, account for garden path effects; or (2) predictability estimates derived from lstm lms fail to capture some aspect of human prediction that is crucial to explaining the processing of garden path sentences. this work investigates the latter possibility. we ask if the gap between the magnitude of human garden path effects in humans and the magnitude that surprisal theory predicts from lms is due to a mismatch between how humans and lms weigh two contributors to word-level surprisal: syntactic and lexical predictability. we hypothesize that the lm next-word prediction objective does not sufficiently emphasize the importance that syntactic structure carries for human readers, who may be more actively concerned with interpreting the sentence. in this scenario, since garden paths are the product of unpredictable syntactic structure-as opposed to an unpredictable lexical item-using a lm predictability estimate for the next word could lead to underestimation of garden path effects. we test the hypothesis that the gap between model and human effects can be bridged by teasing apart the overall predictability of a word from the surprisal associated with the syntactic structure implied by the word (see figure 1 ) and weighting the two factors independently, possibly assigning a higher weight to syntactic surprisal. in this rea- lexical surprisal syntactic surprisal the newfound microbes were... three girls trying to save up for a trip... owls are more flexible... others are unpredictable due to the syntactic structures they imply (trying in girls trying to save up), and are expected to be assigned high syntactic and lexical surprisal. tokens such as microbes in the context the newfound microbes were, on the other hand, appear in a predictable syntactic environment, but are unpredictable due to their low lexical frequency; such words should be assigned low syntactic surprisal but high lexical surprisal. since words that appear in unpredictable syntactic environments are themselves unpredictable, we do not expect to find words with high syntactic surprisal but low lexical surprisal. to compute lstm-based syntactic surprisal, we train the lm with an auxiliary objectiveestimating the likelihood of the next word's supertag under the combinatory categorial grammar (ccg) framework @xcite -following @xcite . such supertags can be viewed as enriched part-of-speech tags that encode syntactic information about how a particular word can be combined with its local environment. we then define syntactic surprisal in terms of the likelihood of the next word's ccg supertag, and propose a method of estimating that likelihood using our modified lms. we validate our formulation of syntactic surprisal by demonstrating that it captures syntactic processing difficulty in garden path sentences, while, crucially, not tracking unpredictability that is due to low frequency lexical items. following van schijndel and linzen (2021), we then use the syntactic and lexical surprisal values derived from those models to predict reading times for three types of garden path sentences. we find that adding syntactic surprisal as a separate predictor does lead to larger estimates of garden path effects, but those estimates are still an order of magnitude lower than empirical garden path effects. finally, we discuss the implications of these findings for surprisal theory and single-stage models of syntactic processing. ## computing syntactic surprisal each incoming word can cause an adjustment in the reader's beliefs about the syntactic structure of the sentence; when a syntactic structure that was assigned a low probability prior to reading the word now has high probability, the word can be said to have high syntactic surprisal. we will operationalize this intuition as the predictability of next word's supertag under the combinatory categorial grammar (ccg) formalism @xcite : we contrast this syntactic surprisal measure with the standard token surprisal measure, which we refer to as lexical surprisal: note that what we call lexical surprisal captures all factors that contribute to a token's predictability, including syntactic ones. in order to compute syntactic and lexical surprisal for a given word, we need models that predict, given a left context, not only the next token, as a standard lm does, but also the next token's supertag. to do this, we train models with both a language modeling and ccg supertagging objective, and estimate the distribution over the next word's tag by marginalizing over the distribution over the next word that is defined by the lm. formally, for a sequence of words w 1 , ..., w @xmath0, ..., c @xmath1, our model estimates the probability of the next word given all observed words, p w n+1 = p (w n+1 | w 1 , ..., w n ), and the probability of the most recent word's supertag given all currently observed words, p cn|@xmath2, ..., w n ). we then infer the distribution over the next word's supertag as if we knew the supertag of the next word c n+1 , we could simply compute the surprisal of that supertag,log p (c n+1 | w 1 , ..., w n ). by contrast with lexical surprisal, however-where there is no uncertainty about the identity of w n+1 once that word has been read-a word's supertag is often ambiguous during incremental processing. consider the verb gathered in the following sentences, for example: in (3), gathered would eventually be assigned the supertag s\np, indicating that gathered is used in its intransitive frame-a number of squirrels assembled together as a group-and takes no direct object. in (4), on the other hand, the appropriate supertag would be (s\np)/np, which indicates that in this sentence gathered is used in a transitive frame and takes the noun phrase a few acorns as a direct object. when processing this sentence incrementally, a reader must maintain this uncertainty over the appropriate supertag for a word past the point at which they have read that word. a measure of syntactic surprisal that aims to model processing difficulty at a particular word should similarly take into account uncertainty over the supertag of a word even after the word itself has been processed. we take this uncertainty into account by using the distribution p cn|wn that our models define over supertags, and computing syntactic surprisal by marginalizing over this distribution: ## validating syntactic surprisal we first validate that our syntactic surprisal measure successfully isolates syntactic predictability from word predictability. to be satisfied that that is the case, we will require two things be true: first, we expect syntactic surprisal to capture processing difficulty that is the result of syntactic unpredictability; and second, we expect that syntactic surprisal is not redundant with lexical predictability. we will evaluate each of these desiderata in turn. ## evaluating against human reading times recall that surprisal theory assumes a linear relationship between surprisal and measures of processing difficulty such as reading times. we follow van @xcite and estimate a mapping between our surprisal measures and reading times by fitting linear mixed-effects models to the filler (i.e., non-garden path) materials from @xcite . in order to compare syntactic and lexical surprisal, we fit four conversion models: one with syntactic surprisal as a predictor, one with lexical surprisal, one with both types of surprisal, and one that does not include either surprisal measure. all four models included baseline predictors other than surprisal -unigram frequency, word position, and word length -which on their own are not expected to capture garden path effects. to account for spillover effects, where processing difficulty from a word spills over to affect reading times at future words, we included all of the aforementioned factors (except word position) not only for the current word but also for the two prior words (a simplification of the technique of van schijndel and linzen 2021). this process is repeated with each of the four sets of surprisals extracted from our four language model/supertagger instances. further details about the surprisal-to-rt conversion process are presented in appendix a.1. after all four of our models have been fit to the filler items, we use the estimated coefficients to predict reading times for the each of the critical items. ## results predicted rt differences from our conversion models, as well as the rt differences observed in humans, are presented in figure 4a . regardless of the predictors used in the mixed-effects model-lexical surprisal, syntactic surprisal, neither, or bothpredicted reading time differences greatly underestimate the reading time differences observed in humans. this is unlikely to be an issue with our surprisal-to-reading-times conversion method more broadly, as at the pre-disambiguation word, rts and predicted rts match much more closely than in post-disambiguation regions (particularly in capturing effects in the pre-disambiguation region in nps sentences), indicating that the difference in magnitudes is due specifically to an underestimation of the garden path effect. while the inclusion of syntactic surprisal does not close the gap between model predictions and the empirical reading times, it does typically lead to a larger predicted garden path effect. to see this difference more clearly, in figure 4b we exclude the human reading times and zoom in on the garden path effects predicted by the models. to determine whether adding syntactic surprisal as a predictor affected the magnitude of the garden path effects we predicted, we fit a linear mixed effects model over all of our conversion models' predicted reading times for each garden path construction at each word in the critical region. we present the results of this analysis for the effect of interest (the interaction between the conversion model and the garden path effect) in ## discussion what is the source of the discrepancy between the magnitude of garden path effects in humans and surprisal-based estimates of those magnitudes from neural network language models? in this paper, we have evaluated one possible answer to this question: that word predictability estimates from lms underweight the importance of syntax to the predictions made by humans. we have proposed a method of estimating syntactic predictability from lstm lms augmented with a ccg supertagging auxiliary objective; confirmed that this measure matches our intuitive desiderata from a syntactic surprisal measure; and compared garden path effect magnitude predictions derived from standard, lexical surprisal and syntactic surprisal. our main finding is that while the syntactic surprisal measure we propose does typically lead to larger predicted garden path effects, model-predicted garden path effects still vastly underestimate the magnitude of garden path effects found in humans. we defined syntactic surprisal in terms of the predictability of the next word's ccg supertag. this choice is motivated by the relative simplicity of computing this measure-a straightforward auxiliary objective that can be added to any conceivable neural language model-as well as two substantive desiderata: first, we would like the measure to capture processing difficulty due to syntactic unpredictability. since a word's ccg supertags captures how the word combines with the local syntactic structure, we hypothesize that the surprisal of that mvrr nps npz p r e -d is a m b d is a m b s p il lo v e r 1 s p il lo v e r 2 p r e -d is a m b d is a m b s p il lo v e r 1 s p il lo v e r 2 p r e -d is a m b d is a m b s p il lo v e r 1 s p il lo v e r 2 0 50 100 150 garden path effect (ms) both surprisals syntactic surprisal only lexical surprisal only no surprisals human rts (a) model predictions and human results. mvrr nps npz p re -d is a m b d is a m b s p il lo v e r 1 s p il lo v e r 2 p re -d is a m b d is a m b s p il lo v e r 1 s p il lo v e r 2 p re -d is a m b d is a m b s p il lo v e r 1 s p il lo v e r 2 -5 0 5 10 15 garden path effect (ms) both surprisals syntactic surprisal only lexical surprisal only no surprisals (b) model predictions only. supertag-which indicates the extent to which that syntactic combination is unexpected-is a good predictor of syntactic unpredictability. this was borne out in our analysis that showed that syntactic surprisal predicts differences in the correct direction in three garden path constructions. second, since syntactic surprisal is designed to isolate syntactic predictability from other forms of predictability, it should not be perfectly correlated with lexical factors. the comparisons to lexical surprisal and word frequency showed that this desideratum was met: we were able to identify in our materials words that were lexically surprising but had low syntactic surprisal, and we found a positive correlation between frequency and syntactic surprisal -the opposite of what would be predicted if high syntactic surprisal was driven by low word frequency. the increase in model-predicted garden path magnitudes when we use syntactic surprisal, compared to using just standard lexical surprisal, suggests that predictability estimates from lstm lms indeed understate the role that syntactic factors play in human prediction. to see why that is, recall that syntactic surprisal captures a subset of the variance that lexical surprisal does. the fact that adding syntactic surprisal produces a better fit to human reading times than lexical surprisal, then, suggests that syntactic factors affect lexical surprisal less than they would need to in order to capture variation in human reading times. one potential explanation for this discrepancy is the difference in disambig mvrr nps npz both vs. syntactic only β = 0.37, @xmath3.001 β = 1.19, @xmath4.001 β = 0.15, @xmath5.056 syntactic only vs. lexical only β = 3.26, @xmath6.001 β = 0.89, @xmath7.001 β = 1.89, @xmath8.001 syntactic only vs. neither β = 6.77, @xmath9.001 β = 2.85, @xmath10.001 β = 1.77, @xmath11.001 spillover 1 mvrr nps npz both vs. syntactic only β = 1.80, @xmath12.001 β = 1.25, @xmath13.001 β = 2.09, @xmath14.001 syntactic only vs. lexical only β = -1.02, @xmath15.001 β = -0.53, @xmath16.001 β = -1.62, @xmath17.001 syntactic only vs. neither β = 5.10, @xmath18.001 β = 3.27, @xmath19.001 β = 2.36, @xmath20.001 spillover 2 mvrr nps npz both vs. syntactic only β = 1.68, @xmath21.001 β = 0.92, @xmath22.001 β = 2.10, @xmath23.001 syntactic only vs. lexical only β = -4.59, @xmath24.001 β = -3.86, @xmath25.001 β = -5.03, @xmath26.001 syntactic only vs. neither β = -0.30, @xmath27.001 β = -0.92, @xmath28.001 β = -3.61, @xmath29.001 the tasks humans and lms perform: while lms need only predict words in corpora, humans must to comprehend what they read. while both tasks demand some sensitivity to syntactic structure, the need to interpret sentences may place greater importance on predicting structure, leading to a higher sensitivity to syntactic unpredictability. while models with syntactic surprisal provided a better fit to the human data than those with just lexical surprisal, there remained a very large discrepancy between model-predicted and human garden path effect sizes. it may be possible to further close this gap within the surprisal framework using different approaches to estimating syntactic predictability; one such approach could rely on recurrent neural network grammars @xcite , which derive word-level predictability estimates from explicit syntactic parsing mechanisms. another possibility is that the discrepancy is not due to flaws in our estimates of human predictability: perhaps surprisal, even based on a perfect simulation of human predictions, is simply not the correct account of the magnitude of the garden path effect observed in humans @xcite . one family of alternative accounts consists of two-stage, serial models of processing @xcite @xcite @xcite . in such a model, when readers first read through the ambiguous fragment of the sentence, they commit to a small set of preferred parses. when they reach a disambiguating region where all of the parses they have committed to are no longer consistent with the input, a reader would engage a separate, costly reanalysis process in order to construct a new partial parse consistent with the all of the currently available input. the processing cost associated with this reanalysis procedures incurs a slowdown in reading times that does not occur in an unambiguous sentence where the incorrect initial parse is not available, resulting the garden path effects that we observe. unlike surprisal-based accounts, however, it is often unclear how to derive broad-coverage, quantitative predictions for the size of garden path effects from existing two-stage accounts. as a result, it is difficult to know whether the quantitative mismatches between surprisal-accounts and human reading times that we observed should be taken as evidence for an explicit reanalysis process. this further highlights the need for precise implementations of two-stage serial models that we can quantitatively evaluate against surprisal accounts. using surprisals generated from four instances of our lm/supertagging model. these models differed only in the random seed used during the initialization and training procedure. in figure 4b in the main text, we presented predicted reading times averaged across these analyses. in figure 5 we present the same results broken out across each model instance.