How does the brain process syntactic structure while listening? (2023)

Abstract: Syntactic parsing is the task of assigning a syntactic structure to a sentence. There are two popular syntactic parsing methods: constituency and dependency parsing. Recent works have used syntactic embeddings based on constituency trees, incremental top-down parsing, and other word syntactic features for brain activity prediction given the text stimuli to study how the syntax structure is represented in the brain’s language network. However, the effectiveness of dependency parse trees or the relative predictive power of the various syntax parsers across brain areas, especially for the listening task, is yet unexplored. In this study, we investigate the predictive power of the brain encoding models in three settings: (i) individual performance of the constituency and dependency syntactic parsing based embedding methods, (ii) efficacy of these syntactic parsing based embedding methods when controlling for basic syntactic signals, (iii) relative effectiveness of each of the syntactic embedding methods when controlling for the other. Further, we explore the relative importance of syntactic information (from these syntactic embedding methods) versus semantic information using BERT embeddings. We find that constituency parsers help explain activations in the temporal lobe and middle-frontal gyrus, while dependency parsers better encode syntactic structure in the angular gyrus and posterior cingulate cortex. Although semantic signals from BERT are more effective compared to any of the syntactic features or embedding methods, syntactic embedding methods explain additional variance for a few brain regions.

## introduction a key assumption in psycholinguistics is that sentence processing involves two operations: (i) the construction of a syntactic structure that represents 1 https://tinyurl.com/brainsyntax the relation between its components and (ii) the retrieval of the meaning of single linguistic units from semantic memory. when presented with a sentence in a task, humans can understand word meaning effectively while reading and listening. listeners and readers appear to extract a similar semantic meaning from narrative stories @xcite , hence suggesting that the brain represents semantic information in an amodal form, i.e., independent of input modality. further, earlier language-fmri encoding studies have observed that sentence semantics alone cannot explain all the variance in brain activity; syntactic information can also be used to explain some of the variance @xcite . prior to different aspects of semantic interpretation, the brain performs syntactic structure analysis inherently @xcite . the syntactic information helps to identify the structural constituents that have to be interpreted as nominal, ordinal, or noun phrases, e.g., we identify "brazil", "four", "world cups", and "2002" in a sentence: "brazil won four world cups till 2002" before interpreting the semantics. hence, investigating how the brain encodes syntactic word features is crucial for understanding language comprehension in the brain. two paradigms of syntactic parsing: constituency and dependency are two different syntactic formalisms using different structural primitives (dependency relations and phrases). there has been some discussion in the field of theoretical linguistics with regard to whether they capture the same information or to what degree the structures they sanction are equivalent @xcite . discussing the linguistic information the two parsers capture, @xcite states from a theoretical linguistic point of view that they describe distinct syntactic entities; thus, they are not strictly equivalent. dependencies capture direct relations between words, identical to thesuch that at each level, each grouping acts as a syntactic unit @xcite . moreover, according to @xcite only dependencies can express the syntactic word-to-word relations of a sentence, whereas constituency expresses the linear order of a sentence. on the other hand, incremental top-down parser processes input words from left to right, producing the possible parses in a top-down manner as future words are read. therefore, @xcite sees the two grammars as complementary but not equivalent. following these last observations, we consider dependency and constituent structures as distinct and the type of information that they capture as nonequivalent. the question we address in this study is whether different brain regions are associated with building different kinds of syntactic structure. we compare the predictive power of syntactic structural measures derived from these parsers with regard to modeling the brain activity in language processing areas recorded during naturalistic story listening. ## feature representations we used four different features computed per word to simultaneously test different syntactic and semantic representations. in contrege complete tree (cc), the largest subtree completed by a given word refers to the subtree with the largest height that also satisfies the following conditions -the given word must be one of its leaves and all of its leaves must only contain words that have been seen till then. in contrege incomplete tree (ci), the embeddings are constructed using incomplete subtrees that are constructed by retaining all the phrase structure grammar productions that are required to derive the words seen till then, starting from the root of the sentence's tree. if incomplete subtrees are more representative of the brain's processes, it would mean that the brain correctly predicts certain phrase structures even before the entire phrase or sentence is read. the incremental top-down parser is a statistical syntactic parser that processes input strings from left to right, producing partial derivations in a top-down manner, using beam search as detailed in @xcite . specifically, we use the implementation as described here foot_1 . the inc embeddings are obtained using exactly the same methods as described in section 3 of (reddy and @xcite . the brain could be computing several possible top-down partial parses that can derive the words seen so far and modifying the list of possible parses as future words are read. the inc feature space is constructed to encode the different possible parse trees that can derive the words seen so far. when considering parse tree based representations, the embeddings may contain information about what is yet to be seen by the subject. however, this is not a problem since it mimics the human capability of guessing what is to come next. with this embedding space, we attempt to measure the ability of the brain to predict future constituents correctly. ## dataset curation brain imaging dataset the "narratives" collection aggregates a variety of fmri datasets collected while human subjects listened to real spoken stories @xcite . we analyze data from 82 subjects listening to the story titled 'pieman' with 282 trs (repetition time -fmri recorded every 1.5 sec.). we chose this story since it contains maximum number of subjects in the "narratives" collection. the dataset is in english and contains 957 words across 67 sentences. the story duration is 07m:02s. we use the multi-modal parcellation of the human cerebral cortex (glasser atlas: consists of 180 rois in each hemisphere) to display the brain maps ( glasser et al., 2016) (v) middle frontal gyrus (mfg: 55b); (vi) inferior frontal gyrus orbital (ifgorb: a47r, p47r, a9-46v), (vii) posterior cingulate cortex (pcc: 31pv, 31pd, pcv, 7m, 23, rsc); and (viii) dorsal medial prefrontal cortex (dmpfc: 9m, 10d, d32) @xcite @xcite . the dataset has been made available freely without restrictions by @xcite . downsampling since the rate of fmri data acquisition (@xmath0.5sec) was lower than the rate at which the text stimulus was presented to the subjects, several words fall under the same tr in a single acquisition. hence, we match the stimulus acquisition rate to fmri data recording by downsampling the stimulus features using a 3-lobed lanczos filter @xcite . after downsampling, we obtain chunk-embedding corresponding to each tr. tr alignment to account for the slowness of the hemodynamic response, we model the hemodynamic response function using finite response filter (fir) per voxel and for each subject separately with 8 temporal delays corresponding to 12 seconds. ## methodology encoding model to explore how and where syntactic and semantic specific features are represented in the brain when listening to stories, we extract dif-ferent features describing each stimulus sentence and use them in an encoding model to predict brain responses. if a feature is a good predictor of a specific brain region, information about that feature is likely encoded in that region. the main goal of each fmri encoder model is to predict brain responses associated with each brain voxel when given stimuli. we train a model per subject separately. following the literature on brain encoding @xcite @xcite @xcite @xcite , we choose to use a ridge regression model instead of more complicated models. we plan to explore more such models as part of future work. the ridge regression objective function for the here, w are the learnable weight parameters, ∥.∥ f denotes the frobenius norm, and λ > 0 is a tunable hyperparameter representing the regularization weight. λ was tuned on a small disjoint validation set obtained from the training set. cross-validation we follow 4-fold (@xmath1. all the data samples from k-1 folds were used for training, and the model was tested on samples of the left-out fold. evaluation metric we evaluate our models using the popular brain encoding evaluation metric, r 2 . let tr be the number of time repetitions. let @xmath2. thus, @xmath3. we use r 2 (y, ŷ ) metric to measure the coefficient of determination for every voxel. we average r 2 score over all voxels in a region to get region-level aggregated metric. finally, they are further averaged across all subjects to obtain final region-level metrics, which are reported with mean and standard deviation. statistical significance we run a permutation test to check if r 2 scores are significantly higher than chance. we permute blocks of contiguous fmri trs, instead of individual trs, to account for the slowness of the underlying hemodynamic response. we choose a standard value of 10 trs. the predictions are permuted within fold 5000 times, and the resulting r 2 scores are used as an empirical distribution of chance performance, from which the p-value of the unpermuted performance is estimated. we also run a bootstrap test, to test if a model has a higher r 2 score than another. in each iteration, we sample with replacement the predictions of both models for a block of trs, compute the difference of their r 2 , and use the resulting distribution to estimate the p-value of the unpermuted difference. finally, the benjamni-hochberg false discovery rate (fdr) correction @xcite is used for all tests (appropriate because fmri data is considered to have positive dependence @xcite ). the correction is performed by grouping all the voxel-level p-values (i.e., across all subjects and feature groups) and choosing one threshold for all of our results. the correction is done this way as we test multiple prediction models across multiple voxels and subjects. ## experiments and results we discuss detailed hyper-parameter settings in appendix a. which word representations are semantic versus syntactic? we first empirically show that syntactic embeddings do not encode a significant amount of semantic information. in particular, we train the ridgecv regression model in a 10-fold cross-validation setting to predict the semantic glove @xcite features (300 dimensions) using syntactic embeddings for all the representations, similar to earlier works @xcite @xcite . average r 2 scores are as follows: bert (0.560), cc (0.052), ci (0.020), dep (0.170), inc (0.040), pd (0.183), cm (0.027), and pu (0.005). these r 2 scores indicate that (a) overall, bert has high semantic information compared to other embeddings, and (b) constituency parsers have low semantic information compared to dep. overall, all the syntactic embeddings consist of very low semantic information. hence, it is reasonable to infer that any additional variance predicted by the syntactic parsing methods compared to the semantic feature space (bert) is mainly due to their syntactic information. ## discussion in this section, we correlate our empirical findings about syntactic parsing methods with previously proposed neuroscience theories. from fig. 4 , we observe that activity in the left temporal lobe (atl and ptl) seems to be predicted well using either cc or basic syntactic (pd) representations. these results are supported by theory of @xcite , who concluded that parts of the ptl are involved in hierarchical lexical-syntactic structure building, while the atl is a knowledge store of entities. while activity in the left ifgorb, left pcc, and left ag seems to be better modeled by basic syntactic feature (pd) representations, that in mfg seems to be related to cc representations. dep embeddings seem to perform better for activity in the left ag, left atl and left ifg. this supports the theory of @xcite , which reports that atl is a knowledge store of entities and ag is a store of thematic relations between entities. a sub-roi in the left ag, namely parietal area g inferior (pgi) has significantly more number of voxels sensitive to dependency features when we control for all other syntactic features. on the other hand sub-rois in the right temporo-parietooccipital junction (tpoj) are more sensitive to incremental top-down syntactic features (appendix fig. 16 ). while it is known that ag is sensitive to stimuli that are connected through a narrative rather than unconnected words @xcite , the current findings suggest that distinct sub-rois within ag are related to different syntactic features. further sub-regions in the prefrontal cortex such as brodmann area (ba) 44 and the inferior frontal junction area (ifja) also seem to be related to representations of dependency parser (appendix fig. 19 ). the results in the prefrontal cortex seem to concur with the observations of @xcite and @xcite who have shown that broca's area (brodmann areas 44 and 45) has higher brain activation while processing complex sentences. since narrative listening also involves processing highly complex sentences, consistent activation found in left brodmann areas 44 and 45 may relate to parsing of sentences or to see if they had distinct meanings. the right hemisphere activation in the language network (ag, atl, ptl, ifg, mfg, ifgorb, pcc, and dmpfc) on the whole seems to be associated with basic syntactic features such as word length, word frequency, word count as embodied in cm representations. in linguistic studies, inc has been shown to be effective in checking if sentences with different syntax, have the same or different meaning. this in line with our observation that representations from inc parser seem to be more related to language regions (inferior frontal gyrus, ifg) in the right hemisphere as shown in fig. 19 . overall, @xcite concluded that syntax processing is not limited to specific regions (left ifg or broca's area). along with ifg, other regions such as ptl, atl, mfg, and if-gorb are also involved in different stages of syntax processing @xcite . our results (fig. 2 ) also seem to support distributed representation of syntax across the language network. moreover, our results clearly show the kind of syntax encoded by these individual rois. ## conclusion we studied the relative importance of multiple constituency and dependency syntax parsing methods for fmri prediction for the listening task. we find that (1) both cc and dep are effective; cc is more important than ci, (2) cc is better in temporal cortex and mfg, while dep is better in ag and pcc, (3) while bert embeddings seem to be the most effective, syntactic embedding methods also explain additional variance for a few rois. in line with previous works, we find that syntax and semantic processing is spread across multiple brain areas.