Lossy Context Surprisal Predicts Task-Dependent Patterns in Relative Clause Processing (2024)

Abstract: English relative clauses are a critical test case for theories of syntactic processing. Expectation- and memory-based accounts make opposing predictions, and behavioral experiments have found mixed results. We present a technical extension of Lossy Context Surprisal (LCS) and use it to model relative clause processing in three behavioral experiments. LCS predicts key results at distinct retention rates, showing that task-dependent memory demands can account for discrepant behavioral patterns in the literature.

## introduction a fundamental goal of computational psycholinguistics is to predict and explain syntactic processing difficulty as manifested in reading times. surprisal from modern language models is a strong predictor of reading times on naturalistic text: words take longer to read when they are less predictable (e.g. @xcite . this finding aligns with expectation-based theories of syntactic processing @xcite . however, surprisal fails to account for certain effects from the psycholinguistic literature -particularly locality effects, in which longer syntactic dependencies lead to increased processing effort (e.g. @xcite . under surprisal theory, this is unexpected: additional intervening context should generally make prediction easier. locality effects are naturally explained in terms of human memory limitations, which motivate memory-based theories of syntactic processing. one example is dependency locality theory @xcite , which posits that the processing cost of integrating a syntactic dependency is proportional to dependency length. similar locality predictions arise from cue-based retrieval theories (e.g. @xcite . ## background measuring incremental processing behavioral methods which track word-by-word reading time (rt) offer scientific insight into human language processing, as longer rts reflect processing difficulty. special eye-tracking (et) equipment can collect rt data in a laboratory setting by monitoring participants' eye movements as they read @xcite . this method most closely approximates natural reading, but et data collection is resourceintensive and the resulting rts can be noisy and challenging to interpret. one crucial source of noise comes from spillover effects: longer processing time for one word can "spill over" to following words. in such cases, systematically longer rts on a specific word do not reflect difficulty processing that word, but instead the word or words preceding. an alternative cost-effective source of rt data is self-paced reading (spr), in which participants must press a button to reveal each word in sequence. unfortunately, spillover effects are typically much larger in spr compared to et data. the maze task @xcite modifies spr by introducing distractors: participants are shown two words at each step, and must select the word which correctly continues the sentence. this task is more cognitively demanding, and appears to reduce or eliminate spillover effects @xcite . @xcite compare maze and et for three types of ambiguous sentences and find that maze rts capture most -but not all -patterns of incremental processing difficulty seen in et rts. in this paper, we consider the possibility that higher working memory demands in the maze task account for key discrepancies between maze and et results. modeling memory and expectation language models (lms) are typically trained on a nextword prediction objective, which aligns them with the expectation-based account of surprisal theory. modern large language models, however, have become worse predictors of human rt data due to their superhuman capacity for memorization @xcite . this has motivated modeling approaches which combine lms with memory constraints. @xcite propose a model architecture with a single self-attention head, which reduces the capacity to retrieve earlier representations from context. @xcite find improved fits to rts by simply truncating words from the preceding context. here, we model memory constraints with resource-rational lossy context surprisal (rr-lcs; @xcite , which learns to stochastically retain or delete specific words from the representation of the preceding context. crucially, we can systematically vary the lcs retention rate to simulate different patterns of working memory engagement. ## modeling relative clause processing the increased difficulty in processing object relative clauses (orcs) compared to subject relative clauses (srcs) provides a testing ground for effects of memory and expectation. memory-based accounts such as dependency locality theory (dlt; @xcite predict increased reading time (rt) at the orc verb, reflecting integration of long-distance dependencies. this prediction has been realized in eye-tracking (et) studies @xcite . the expectation-based surprisal theory @xcite , however, predicts an rt slowdown only at the start of the orc noun phrase, and this pattern has been found in maze studies @xcite . vani et al. suggest that the orc verb slowdown found in eyetracking studies may reflect spillover effects rather than memory constraints. we explore the alternative hypothesis that et experiments impose lower memory demands relative to the maze task. at lower retention rates, lossy context surprisal (lcs) models memory con-20% 60% vani et al. ( 2021) staub (2010 straints, but not spillover effects; if lcs captures the patterns in et behavioral data, this supports the interpretation that orc verb slowdowns are memory-driven but also modulated by task demands. using our lcs implementation with subword tokenization, we generate predictions on critical rc stimuli and compare them to behavioral results from maze @xcite and eyetracking @xcite . we draw on roland et al.'s statistical analysis to further distinguish spillover and memory effects. ## discussion our main finding is that low-retention lcs reproduces key predictions of memory-based accounts, and provides a plausible fit to et data -whereas high-retention lcs reproduces expectation-based predictions, and better fits maze data. the maze task requires that participants actively reject distractor words and select the correct sentence continuation; this activity strikes us as clearly more cognitively demanding than naturalistic reading, so task-dependent memory demands present a viable explanation for these discrepant results. 13 an alternative hypothesis suggested by @xcite attributes the orc verb slowdown seen in et data to spillover effects. our analysis indicates that this is unlikely: the orc verb slowdown is consistently predicted by low-retention lcs, pointing toward a memory-driven explanation. to be clear, we do not claim that spillover has no systematic influence on relative clause processing. the detailed modeling analysis conducted by @xcite indicates that spillover at least partly accounts for the orc verb slowdown. the slowdown effect persists, however, even after adjusting for spillover, and our lcs simulations suggest that the slowdown reflects memory constraints (figure 5 ). we note that lcs consistently predicts some patterns which have not been given a formal theoretical articulation. further investigation is required to assess when these discrepancies could be systematic and theoretically meaningful. the orc noun speed-up presents an interesting case study: this effect is not directly predicted by either expectation or memory accounts, but it appears robustly in both lcs predictions and the et data for experiment 1 of @xcite . this unexpected concordance suggests that memory constraints may also drive this effect. on the other hand, lcs appears to incorrectly predict an orc slowdown at the rc np for the @xcite stimuli (figure 5 ); however, closer analysis reveals that this effect is driven by the orc slowdown at the determineron the rc noun itself, lcs once again predicts an orc speedup, and this effect is larger at the lower retention rate of 20%. 14 under lcs, memory constraints appear to drive both the orc verb slowdown and the orc noun speedup, although to our knowledge the latter effect has not been discussed in connection with memory-based accounts. exploring the nature of this connection could be a promising direction for future research. future work could also explore alternative approaches to modeling expectation. while surprisal theory is well-represented in the research literature and closely aligned with the standard language model learning objective, other research has formulated expectation in terms of information gain (e.g. @xcite . under an information gain account, the incremental cost of processing a given word reflects not its conditional probability (as posited by surprisal theory), but rather the uncertainty reduction it provides between alternative sentence continuations. chen and hale (2021) use one such approach, namely entropy reduction @xcite , to model the same relative clause processing asymmetry addressed here. they use corpus statistics to compute word-by-word transitions in entropy over the probabilities of following syntactic derivations, and find that this measure predicts the observed orc slowdown at both the rc np determiner and the rc verb. their model can therefore account for the orc verb slowdown observed in et data -however, it would not appear to predict the pattern observed in maze data by @xcite . an alternative information gain approach (e.g. hoover, 2024) could in principle address such task-dependent effects. in the meantime, we note that lcs straightforwardly captures this variation in relative clause processing as a consequence of memory demands. other avenues for future research could address further limitations of the current study. for instance, it might be more appropriate to vary retention rates not only at the experiment level, but also to model differences between individual participants. one could also pursue more interpretability in lcs predictions through detailed analysis of specific word-level reconstructions. lastly, this paper focuses on one grammatical phenomenon in one language; a thorough treatment of memory effects in online language comprehension will naturally require a broader scope of evaluation. ## conclusion we find that manipulating the retention rate of a lossy context surprisal (lcs) model captures taskdependent differences observed in reading times (rts). filler item rts from the maze task are best fit with a relatively high retention rate (e.g. 60%), while lower retention (20%) better predicts eyetracking rts for those same items. furthermore, based on these task-dependent retention rates, lcs correctly predicts critical rt patterns observed for english relative clauses. in particular, lowretention (20%) lcs follows memory-based theories and predicts higher rts for object relative verbs -an effect found in eye-tracking but not maze studies. these results can explain the apparently contradictory behavioral evidence supporting both memory-and expectation-driven accounts: relative clause processing is likely modulated by the memory demands of the task, and we can use lcs to model this phenomenon.