Contributions of Propositional Content and Syntactic Category Information in Sentence Processing (2021)

Abstract: Expectation-based theories of sentence processing posit that processing difficulty is determined by predictability in context. While predictability quantified via surprisal has gained empirical support, this representation-agnostic measure leaves open the question of how to best approximate the human comprehender’s latent probability model. This work presents an incremental left-corner parser that incorporates information about both propositional content and syntactic categories into a single probability model. This parser can be trained to make parsing decisions conditioning on only one source of information, thus allowing a clean ablation of the relative contribution of propositional content and syntactic category information. Regression analyses show that surprisal estimates calculated from the full parser make a significant contribution to predicting self-paced reading times over those from the parser without syntactic category information, as well as a significant contribution to predicting eye-gaze durations over those from the parser without propositional content information. Taken together, these results suggest a role for propositional content and syntactic category information in incremental sentence processing.

## introduction much work in sentence processing has been dedicated to studying differential patterns of processing difficulty in order to shed light on the latent mechanism behind online processing. as it is now wellestablished that processing difficulty can be observed in behavioral responses (e.g. reading times, eye movements, and event-related potentials), recent psycholinguistic work has tried to account for these variables by regressing various predictors of interest. most notably, in support of expectationbased theories of sentence processing (hale, 2001; levy, 2008), predictability in context has been quantified through the information-theoretical measure of surprisal (shannon, 1948). although there has been empirical support for n-gram, pcfg, and lstm surprisal in the literature (goodkind and bicknell, 2018; hale, 2001; levy, 2008; shain, 2019; smith and levy, 2013), as surprisal makes minimal assumptions about linguistic representations that are built during processing, this leaves open the question of how to best estimate the human language comprehender's latent probability model. one factor related to memory usage that has received less attention in psycholinguistic modeling is the influence of propositional content, or meaning that is conveyed by the sentence. early psycholinguistic experiments have demonstrated that the propositional content of utterances tends to be retained in memory, whereas the exact surface form and syntactic structure are forgotten (bransford and franks, 1971; jarvella, 1971). this suggests that memory costs related to incrementally constructing a representation of propositional content might manifest themselves in behavioral responses during online sentence processing. in addition, there is evidence suggesting that parsing decisions are informed by the ongoing interpretation of the sentence (brown-schmidt et al., 2002; tanenhaus et al., 1995). based on this insight, prior cognitive modeling research has sought to incorporate propositional content information into various complexity metrics. a prominent approach in this line of research has been to quantify complexity based on the compatibility between a predicate and its arguments (i.e. thematic fit, baroni and lenci 2010, chersoni et al. 2016, padó et al. 2009). however, these complexity metrics can only be evaluated at a coarse per-sentence level or at critical regions of constructed stimuli where predicates and arguments are revealed, making them less suitable for studying online processing. a more distribu-a) lexical attachment decision ℓ t b) grammatical attachment decision g t unlike these models, our approach seeks to incorporate propositional content by augmenting a generative and incremental parser to build an ongoing representation of predicate context vectors, which is based on a categorial grammar formalism that captures both local and non-local predicateargument structure. this processing model can be used to estimate per-word surprisal predictors that capture the influence of propositional content differentially with that of syntactic categories, which are devoid of propositional content. @xcite our experiments demonstrate that the incorporation of both propositional content and syntactic category information into the processing model significantly improves fit to self-paced reading times and eye-gaze durations over corresponding ablated models, suggesting their role in online sentence processing. in addition, we present exploratory work showing how our processing model can be utilized to examine differential effects of propositional content in memory-intensive filler-gap constructions. @xcite note that this distinction of propositional content as retained information about the meaning of a sentence and syntactic categories as unretained information about the form of a sentence may differ somewhat from notions of semantics and syntax that are familiar to computational linguists -in particular, predicates corresponding to lemmatized words fall on the content side of this division here because they are retained after processing, even though it may be common in nlp applications to use them in syntactic parsing. ## background the experiments presented in this paper use surprisal predictors calculated by an incremental processing model based on a probabilistic left-corner parser (johnson-laird, 1983; van schijndel et al., 2013). this incremental processing model provides a probabilistic account of sentence processing by making a single lexical attachment decision and a single grammatical attachment decision for each input word. @xcite surprisal can be defined as the negative log of a conditional probability of a word w t and a state q t at some time step t given a sequence of preceding words w 1..t-1 , marginalized over these states: these conditional probabilities can in turn be defined recursively using a transition model: a probabilistic left-corner parser defines its transition model over possible working memory store states q @xmath0, . . . , a d t /b d t , each of which consists of a bounded number d of nested derivation fragments a d t /b d t . each derivation fragment spans a part of a derivation tree below some apex node a d t , lacking a base node b d t yet to come. at each time step, the parser generates a lexical attachment decision ℓ t , a word w t , a grammatical at- figure 2 : lambda calculus expression for the propositional content of the sentence many people eat pasta, using generalized quantifiers over discourse entities and eventualities. as shown in figure 1 , the lexical attachment decision ℓ t generates a new complete node a ℓ t based on (m ℓ t ) whether the word matches the base of the most recent derivation fragment; and the grammatical attachment decision g t generates a new derivation fragment a g t /b g t based on (m g t ) whether the parent of a grammar rule with this new complete node as a left child matches the base of the most recent remaining derivation fragment. the semantic processing model described in this paper extends the above left-corner parser to incorporate propositional content by conditioning lexical and grammatical decisions on sparse vectors of predicate contexts h a d t and h b d t in addition to category labels c a d t and c b d t in apex and base nodes a d t and b d t . these predicate context vectors for nodes in a derivation tree of a sentence can be defined in terms of argument positions of variables signified by these nodes in predicates of a logical form translation of that sentence. for example, in figure 2 , the variable e 2 (signified by the word eat) would have the predicate context eat 0 because it is the zeroth (initial) participant of the predication (eat e 2 x 1 x 3 ). @xcite similarly, the variable x 3 would have both the predicate context pasta 1 , because it is the first participant (counting from zero) of the predication (pasta e 3 x 3 ), and the predicate context eat 2 , because it is the second participant (counting from zero) of the predication (eat e 2 x 1 x 3 ). these predicate contexts are obtained by reannotating the training corpus using a generalized categorial grammar of english (nguyen et al., 2012), which is sensitive to syntactic valence and non-local dependencies. lexical attachment probabilities. the probability of each lexical decision ℓ t in this parser is therefore decomposed into one term for generating a match decision m ℓ t and a predicate context vector h ℓ t , and another term for generating a syntactic category label c ℓ t for the new complete node a ℓ t : the probability of generating the match decision and the predicate context vector depends on the base node b d t-1 of the previous derivation fragment: word probabilities. probabilities for generating words are estimated as the probability of generating their character sequence using a recurrent neural network implementation of a character model. grammatical attachment probabilities. the probability of each grammatical decision g t in this parser is similarly decomposed into a term for generating a match decision m g t and a composition operator for a grammar rule o g t , @xcite and terms for category labels c g t and c g t at the apex and base nodes of the new derivation fragment: the probability of generating the match decision and the composition operator depends on the base node of the previous derivation fragment and the new complete node a ℓ t : and sparse composition matrices b o g t , which can be used to compose predicate context vectors associated with the base node b g t of the new derivation @xcite examples of composition operators include using the predicate context of the left child as a modifier or an argument, as well as introducing or discharging filler-gap dependencies. fragment (see figure 4 ): these composition matrices allow predicate contexts to propagate appropriately through the tree to allow parsing decisions to depend on predicates that may be several words away. resulting store state probabilities. in order to update the store state based on the lexical and grammatical decisions, derivation fragments above the most recent nonterminal node are carried forward, and derivation fragments below it are set to null (⊥), ## isolating content and category contributions in order to examine the contribution of propositional content on the content-sensitive processing model, the model is modified to allow it to be trained to make lexical and grammatical decisions without conditioning on the predicate context vec-tors, these two ablated models will respectively be referred to as the content-and category-ablated models in the following experiments. ## experiment 2: self-paced reading in order to evaluate the contribution of propositional content and syntactic categories to predicting behavioral responses, surprisal predictors were calculated from the content-sensitive processing model and its two ablated versions, which are outlined in section 3. subsequently, linear mixedeffects models containing common baseline predictors and one or more surprisal predictors were fitted to self-paced reading times. finally, a series of likelihood ratio tests (lrts) were conducted in order to evaluate the contribution of the surprisal predictor from the full processing model to regression model fit. ## experiment 3: eye-tracking data in order to examine whether the results observed in experiment 2 generalize to other latency-based measures, linear-mixed effects models were fitted on the dundee eye-tracking corpus (kennedy et al., 2003). following similar procedures to experiment 2, a series of lrts were conducted to test the contribution of propositional content and syntactic category information. ## experiment 4: filler-gap constructions observing that surprisal from the full processing model did not contribute significantly to predicting broad-coverage self-paced reading times on top of its content-ablated counterpart in experiment 2, we focus on filler-gap constructions, @xcite in which information about the extracted object is thought to strongly influence the processing of the verb. in order to explore the extent to which integration costs associated with filler-gap constructions could be explained by the influence of propositional content, a series of lrts were conducted to assess the contribution of surprisal from the full processing model to predicting reading times of object-extracted verbs. ## conclusion this paper presents a generative and incremental content-sensitive processing model which factors the contribution of propositional content and syntactic category information. this model can be cleanly ablated to calculate surprisal predictors that differentially isolate the influence of the two components. subsequent experiments demonstrate the utility of both components in predicting human behavioral responses; the inclusion of propositional content resulted in significantly better fits to broadcoverage eye-gaze durations and self-paced reading times of object-extracted verbs. additionally, the inclusion of syntactic category information significantly improved fits to both broad-coverage selfpaced reading times and eye-gaze durations. taken together, these results suggest a role for propositional content and syntactic category information in incremental sentence processing.