Multipath parsing in the brain (2024)

Abstract: Humans understand sentences word-by-word, in the order that they hear them. This incrementality entails resolving temporary ambiguities about syntactic relationships. We investigate how humans process these syntactic ambiguities by correlating predictions from incremental generative dependency parsers with timecourse data from people undergoing functional neuroimaging while listening to an audiobook. In particular, we compare competing hypotheses regarding the number of developing syntactic analyses in play during word-by-word comprehension: one vs more than one. This comparison involves evaluating syntactic surprisal from a state-of-the-art dependency parser with LLM-adapted encodings against an existing fMRI dataset. In both English and Chinese data, we find evidence for multipath parsing. Brain regions associated with this multipath effect include bilateral superior temporal gyrus.

## introduction a major unsolved problem in computational psycholinguistics is determining whether human sentence comprehension considers a single analysis path 1 at a time, or whether it sometimes entertains multiple lines of reasoning about the structure of a sentence. these lines of reasoning typically correspond to syntactic ambiguities (e.g. between subject and modifier as shown in figure 1 ). multipath accounts of ambiguity resolution (e.g. @xcite are quite different from single-path accounts (e.g. @xcite . the debate within cognitive science has focused on the special case of garden path sentences leaving open the broader question: is human comprehension ever multipath in the sense of ranked parallel parsing? if it is, where in the brain does such multipath parsing happen? 1 the "paths" terminology (vs "serial" or "parallel") serves to categorize the cognitive issue as one of process rather than architecture (see @xcite ). the desert sand we take up the call to answer this question (e.g. @xcite by combining broad coverage incremental parsing (e.g. @xcite with information-theoretical complexity metrics to derive predictions about neuroimaging. analyzing an entire book @xcite makes it possible to examine -for the first timeall of the ambiguities that are implicated by a given conception of syntax and a given parsing strategynot just those attested in classic garden path materials @xcite @xcite . we extend the state of the art in incremental generative dependency parsing by using a large language model (bloom; le @xcite together with parameter efficient fine-tuning using adapters @xcite . we fix the parsing strategy while varying the number of allowable analysis paths and connect neural data with parser actions via surprisal (for a review of this information-theoretical metric see @xcite or neuroimaging studies such as @xcite @xcite . this sets up a contrast between a multipath model that considers (at most) five paths at a time, versus a single-path model that can only consider one at a time. the results, reported in section 5, ultimately support the multipath view, as the surprisals from the five-way parser are bettercorrelated with the neuroimaging data. this obtains for both english and chinese. @xcite , further characterized below in §3.4). apart from these cognitive considerations, a complementary motivation for this research is to investigate the parsing improvements that come by leveraging large language models (llms; see accuracy scores in instead of deriving a processing complexity metric directly from the output of a large language model, or correlating llm internal representations with fmri data, as in @xcite and @xcite , this work uses initial substring encodings from an llm to inform an incremental parser. our project also differs from @xcite , who probe llm representations with a view towards inferring a single, unlabelled dependency analysis. in contrast, we use llm representations to score the set of all possible alternative analyses. our scientific goal is to relate such parser states to human brain states, as observed via neuroimaging. ## results both english and chinese paired t-tests on r 2 increase maps show differences in the superior tempo-ral gyrus (stg). the chinese results further show differences in middle temporal gyrus (mtg), while the english results show differences in the parietal lobe, including bilateral angular gyrus. the full list of statistically significant clusters corresponding to figures 7 and 8 can be found in the appendix. although the paired t-tests were twotailed, all of the resulting significant clusters show greater r 2 increase for syntactic surprisal at @xmath0. the r 2 increase maps for each model individually can be found in the appendix. ## discussion multipath parsing effort, outside of the 1-best analysis, localizes to superior temporal regions of the brain bilaterally in both english and chinese. this finding converges with @xcite . crabbé and colleagues parse phrase structure, rather than dependencies, employ a different complexity metric, and use a very large beam (400). despite all these differences, their results implicate roughly the same brain regions. these results also cohere with proposals regarding the large-scale organization of language processing in the brain. on the neuroanatomical pathway model, for instance, "most basic syntactic processes" are handled by a ventral pathway that passes through stg regions identified here @xcite . @xcite similarly point to the ventral pathway as essential for "basic" syntactic processes. this heuristic notion of basic syntactic processing aligns well with stanford dependencies' avowed goal of providing a simple, surface-oriented notation for local gram- ## conclusion and future work the main conclusion is that human parsing is multipath. this follows from observing greater r 2 increase for multipath surprisal than single-path surprisal in fmri data. even with a very bottom-up strategy, such as the arc-hybrid system used here, it seems that more than one path must be pursued in order to best-align with humans' word-by-word effort profile. this conclusion is consistent with disjunctive representations of choices such as pp attachment @xcite . the multipath interpretation that we offer here can be confirmed or refuted with other linked linguistic and neuro-cognitive databases -for in-stance, in different genres or languages. in addition, using brain data with a higher temporal resolution, such as meg, may provide a benefit given the temporary nature of the syntactic ambiguities included in our model. although we take the primary finding to be one of commonality across stg regions in both languages, english listeners did uniquely show an additional effect of multipath parsing in parietal regions. activation in this area has been correlated with measures of human memory such as digit span @xcite . individuals' memory span may modulate the number of paths that they pursue during comprehension @xcite . future work should address the interaction between disambiguation and memory (e.g. @xcite .