Hierarchical Processing of Visual and Language Information in the Brain (2022)

Abstract: In recent years, many studies using deep learning have been conducted to elucidate the mechanism of information representation in the brain under stimuli evoked by various modalities. On the other hand, it has not yet been clarified how we humans link information of different modalities in the brain. In this study, to elucidate the relationship between visual and language information in the brain, we constructed encoding models that predict brain activity based on features extracted from the hidden layers of VGG16 for visual information and BERT for language information. We investigated the hierarchical characteristics of cortical localization and representational content of visual and semantic information in the cortex based on the brain activity predicted by the encoding model. The results showed that the cortical localization modeled by VGG16 is getting close to that of BERT as VGG16 moves to higher layers, while the representational contents differ significantly between the two modalities.

## introduction in recent years, many studies have been conducted to elucidate the information representation mechanisms of the human brain using deep learning. studies using convolutional neural networks (cnns) have confirmed the hierarchical processing of visual information in the brain @xcite . in addition, studies using deep learning models that deal with language have confirmed that it is possible to model the representation of semantic information in the brain @xcite . however, most studies are conducted separately, and the similarities and differences in the brain information representation of both modalities have not been sufficiently discussed. with this background, the objective of this study is to investigate on how the information localization and representation of both modalities are related to each other in the brain -we particularly aim to investigate the hierarchical characteristics of the cortical localization and representation contents of visual and language information in the cerebral cortex by using representational similarity analysis (rsa) @xcite . ## related research in pioneering work in modeling brain representations using deep learning, @xcite showed that there is homology between hierarchical information representations in the human cortex under visual stimuli and those in cnns, and @xcite showed that complexity gradually increases with higher layers in hierarchical processing. in a study using functional magnetic resonance imaging (fmri) and magnetoencephalography, @xcite used deep learning model to show that spatio-temporal dynamics in the human brain cortex during visual object recognition is a hierarchical response. @xcite have revealed the functional organization of the visual cortex of the human brain by analyzing brain activity with the aid of a deep learning model. @xcite introduced the brain hierarchy score, which indicates the degree of hierarchical response based on encoding and decoding to brain activity, and discussed what kind of deep learning models accurately represent the structure of the visual cortex of the human brain, showing that deep learning models with high accuracy in image identification do not necessarily represent the behavior of the visual cortex of the human brain. on the other hand, in a study that models brain representations from semantic features of language, @xcite used fmri to observe brain activity of subjects watching a two-hour natural video and labeled them using 1705 wordnet @xcite -based categories for objects and actions in the video, showing that these categories are not represented in specific brain regions but as locations in a continuous semantic space. @xcite constructed semantic maps in brain regions from brain activity induced by natural speech stimuli, and found that in most regions of the semantic system, there are specific semantic regions and groups of related concepts. @xcite clarified that quantitative modeling of meaning using word2vec @xcite and other methods is an effective means of estimating language activity in the brain through comparison with semantic structures evaluated from human behavior. @xcite introduced lstm @xcite to extract vectors for each word, used them in their encoding model, and achieved more accurate estimation than conventional models. in recent years, the construction of computational models that explain language processing properties in the brain using distributed semantic representations has played an important role. in this context, @xcite scrutinized the still unexplored relationship between the brain representation of sentences and distributed representations, and whether the linguistic features captured by distributed representations can better explain the correlation between brain activities in which sentences are given as linguistic stimuli, and showed the characteristics of distributed representations and their effectiveness. most of the above studies have explored the properties of visual and semantic brain processing separately. therefore, the hierarchical processing from visual to semantic information in the brain is not well understood. in this study, we construct and compare encoding models based on these two different modalities, and investigate the characteristics of information localization and information representation content in the hierarchical processing of visual and semantic information. to predict the brain activity from the features extracted by those deep learning models, we construct encoding models using ridge linear regression. then, to investigate the hierarchical characteristics of cortical localization and representational contents of visual and linguistic information on the cerebral cortex, we apply rsa to analyzing the brain states predicted by the encoding models. ## experiments we have conducted the following three experiments to investigate whether or not:(i) predictable brain regions are similar to both vision and language stimuli; (ii) cortical localization patterns are similar; (iii) representational content is similar. the numbers on the right side of figure 1 correspond to the numbers of the experiments. ## conclusions in this study, we have investigated the the hierarchical characteristics of cortical localization and representational content of visual and linguistic information on the cerebral cortex by means of rsa using prediction accuracy and contents. as a result, in the analysis of cortical localization using prediction accuracy, we found that vgg16, i.e., cnns dealing with image features, was able to model the hierarchy in the cortical localization in the brain, and as it moved from lower to higher layers, it was able to predict brain regions closer to those predicted by bert, i.e., dnns dealing with linguistic features. furthermore, in the analysis of information representation content with predicted brain activity, it was found that the higher layers of vgg16 can model complex cortical localization patterns in the cortex as well as bert. however, we found a large gap between vgg16 and bert in the comparison of the representational contents between the layers. these results suggest that visual information is represented in the same brain regions as linguistic information as it becomes more complex (e.g., category selection regions in the temporal cortex), but even within the same brain regions, there are significant differences between visual and linguistic information, and that modeling with vgg16 and bert alone is not sufficient to fill in these differences. when cortical localization is similar between different modalities, we generally tend to conclude that the representational contents between them are also similar, but the results of this study suggest that the similarity relationship does not necessarily hold, which has an important message to encourage rethinking of the results of previous studies that tackle to elucidate brain information representation for different modalities based solely on the prediction accuracy of brain activity information. based on this, we intend to continue to elucidate the characteristics between modalities.