Is the Brain Mechanism for Hierarchical Structure Building Universal Across Languages? An f MRI Study of C hinese and E nglish (2022)

Abstract: Evidence from psycholinguistic studies suggests that the human brain builds a hierarchical syntactic structure during language comprehension. However, it is still unknown whether the neural basis of such structures is universal across languages. In this paper, we first analyze the differences in language structure between two diverse languages: Chinese and English. By computing the working memory requirements when applying parsing strategies to different language structures, we find that top-down parsing generates less memory load for the right-branching English and bottom-up parsing is less memory-demanding for Chinese.Then we use functional magnetic resonance imaging (fMRI) to investigate whether the brain has different syntactic adaptation strategies in processing Chinese and English. Specifically, for both Chinese and English, we extract predictors from the implementations of different parsing strategies, i.e., bottom-up and top-down. Then, these predictors are separately associated with fMRI signals. Results show that for Chinese and English, the brain utilizes bottom-up and top-down parsing strategies separately. These results reveal that the brain adopts parsing strategies with less memory processing load according to different language structures.

## introduction a hallmark of human language ability is combining linear sequential word inputs into a hierarchical structure using abstract syntactic rules. this ability enables us to create infinite expressions from finite words. previous studies have shown that several brain regions are involved in building the hierarchical syntactic structure @xcite . however, the parsing strategies that the brain uses to build such structures are less discussed. moreover, most of these studies only have been conducted on one language. it is still unknown whether the neural basis of such structures is universal across languages. this paper investigates the neural basis of syntactic structure-building, particularly the parsing strategies that the brain uses, in two diverse languages: chinese and english. we focus on two parsing strategies: top-down and bottom-up. since existing work has demonstrated that a parsing strategy requires different memory space when processing sentences with different branching directions @xcite , we first analyze the branching structures and the memory load generated by two parsing strategies for chinese and english. results show that the dominant branching directions and the memory space required by parsing strategies are both different between the two languages. for the right-branching english, the top-down parsing needs less memory space than bottom-up parsing. whereas for chinese, which is more mixed in branching directions, the bottom-up parsing is less memory-demanding than the top-down parsing. therefore, we have two hypotheses about how the brain processes different languages: to test these two hypotheses, we associate the complexity predictors derived from different parsing strategies with the brain imaging data collected when native speakers were listening to stories. the complexity predictors are the number of parsing operations when using a parsing strategy to integrate each word into the tree. the key assumption is that brain regions engaged in syntactic structurebuilding would show increased activity as the number of parsing operations increases. therefore, if a brain region builds trees following a parsing strategy, then the complexity predictors of this strategy would be able to predict the activation of this brain region. by comparing the prediction performances of different predictors, we can evaluate which parsing strategy better accounts for the brain activity in chinese and english. from the comparative study, we have the following interesting findings: the dominant predictor for chinese is bottom-up but for english it is topdown, which is consistent with the less-memorydemanding strategies for each language. the brain regions with significant effects are also different between chinese and english. however, in further analysis, we find that the data size gap and the correctness of constituency trees both contribute to the brain-region differences. these results support the second hypothesis that the brain adopts parsing strategies with less cognitive load for different languages. in conclusion, our main contributions include: ## related work building hierarchical syntactic structures is an important sub-process of language understanding. existing work that has investigated this sub-process can be categorized into two groups. one is often called controlled experiments that design artificial stimuli to separate the brain activation, such as comparing structured complex sentences or phrases with word lists @xcite @xcite @xcite . these studies have many valuable findings and have identified several brain regions, including the left temporal lobe and the left frontal lobe, that are involved in syntactic structure building @xcite . however, as designing artificial stimuli can only separate situations with or without structurebuilding, these controlled paradigms can hardly be used to further study the parsing strategies that the brain uses to build syntactic structures. as a complement to controlled experiments, another line of work used naturalistic experimental paradigms and explored the brain mechanism using encoding models @xcite @xcite . by represent the structure-building process with syntactic predictors, @xcite explored how the brain builds syntactic structure by using encoding models to predict brain activation with these predictors. this paradigm using syntactic predictors can conveniently incorporate different languages into the same framework. but most of these studies only focused on one language, mainly english @xcite @xcite . less attention has been paid to other languages and cross-language comparisons. moreover, little work has investigated the underlying reasons that drive the brain to utilize a parsing strategy. this paper follows the naturalistic experimental paradigm and aims to explore the structural differences between chinese and english and whether these differences drive the brain to use different parsing strategies in structure-building. ## materials the english and chinese fmri datasets we use were both collected when native speakers were listening to narrative stories. all these audio stories are naturalistic stimuli and highly representative of the language that humans encounter in everyday life. english fmri data the english fmri data we use comes from zhang et al. (2020) foot_0 , which was collected from 19 human subjects. the stimuli included 52 stories downloaded from the moth radio hour foot_1 and each story lasts from 4 to 13 minutes. chinese english left right left right embedded 3882 7258 300 712 total 8995 15072 1744 24381 percent(%) 43.16 48.16 17.20 2.92 while being scanned for fmri, each subject listened to a subset of the audio stories. in total, the story stimuli include 47,356 words, and the vocabulary size is 5,228 words (duplicates were excluded). chinese fmri data we collected the chinese fmri data from 12 chinese native speakers when they were listening to a total of 60 stories. each of the subjects listened to all 60 stories, and each story was listened to once by one subject. during the scanning of fmri, subjects were instructed to stay still and pay attention to the story they were hearing. all stories were downloaded from the renmin daily review website 3 and each of them lasts from 4 to 7 minutes. the 60 stories contain 52,269 words, forming a vocabulary of 9,153 words. this fmri dataset is publicly available at https:// openneuro.org/datasets/ds004078 . more details about fmri acquisition and technical validation can be found in @xcite . both the chinese and english fmri data were preprocessed following the hcp pipeline @xcite . ## language structure analysis chinese and english are two very diverse languages and differ in many aspects. we focus on the branching direction, which is directly correlated with the tree structure. ## fmri experiments to test the two hypotheses, whether the brain uses the same parsing strategy regardless of language structures or whether the brain chooses a strategy with less cognitive load when processing different structures, we conducted an fmri experiment as follows. the overall framework is shown in figure 4 we first computed the syntactic predictors from the parsing process of different parsing strategies. then, we trained the voxel-wise encoding models to predict the fmri signals from these syntactic predictors and compare their prediction accuracy in different brain regions and languages. the methodology for english and chinese is the same, which makes the results of these two languages more comparable. ## results after brain encoding and the significance test, figure 5b and figure 5c are the brain regions that can be significantly predicted by different node count predictors. ## analysis in this section, we investigate the possible reasons for the cross-language differences in the correlated brain regions. our analysis is conducted on two experimental aspects, including the data size and the correctness of constituency trees between chinese and english. ## conclusion to investigate whether the brain mechanism for hierarchical structure building is universal across languages, this work investigated the correlation between language branching directions, parsing strategies, and brain activation. by comparing the fitness of the complexity metrics extracted from different parsing strategies in two diverse languages, i.e., chinese and english, we find experimental results supporting the hypothesis that the language structure may play an important role in determining the parsing strategy that the brain uses. that is, the brain may use different parsing strategies for different language structures to reduce the cognitive load. our results demonstrate the flexibility of the brain mechanism for language processing and highlight the importance of cross-language studies in studying the brain language comprehension.