Unveiling Multi-level and Multi-modal Semantic Representations in the Human Brain using Large Language Models (2024)

Abstract: In recent studies, researchers have used large language models (LLMs) to explore semantic representations in the brain; however, they have typically assessed different levels of semantic content, such as speech, objects, and stories, separately. In this study, we recorded brain activity using functional magnetic resonance imaging (fMRI) while participants viewed 8.3 hours of dramas and movies. We annotated these stimuli at multiple semantic levels, which enabled us to extract latent representations of LLMs for this content. Our findings demonstrate that LLMs predict human brain activity more accurately than traditional language models, particularly for complex background stories. Furthermore, we identify distinct brain regions associated with different semantic representations, including multi-modal vision-semantic representations, which highlights the importance of modeling multi-level and multi-modal semantic representations simultaneously. We will make our fMRI dataset publicly available to facilitate further research on aligning LLMs with human brain function.

## introduction language models, which learn the statistical structure of languages from large corpora to enable machines to understand semantics, have made significant advances @xcite @xcite . because semantic comprehension is fundamental to human intelligence, the correspondence between human brain activity and the latent representations of language models has been an intriguing subject of research. to examine the correspondence between human and language models, in recent studies, researchers used brain encoding models @xcite @xcite * equal first author. the quantitative assessment of the correspondence between the two can serve as a unique benchmark for modern large language models (llms) because it potentially provides a biological benchmark for the alignment between llms and humans. in previous studies, researchers typically focused on a single aspect of semantic comprehension (e.g., speech content); however, realistic scenarios are inherently multifaceted. for instance, a scene in which two people are speaking can be depicted through multiple semantic levels in language: their speech content, their identities and the visual appearance of their outfits, the location and time of the scene, and the broader context of the conversation. in traditional studies, researchers often address these levels of semantics separately, which leads to a lack of clarity on how multiple levels of semantic content are attributed to brain activity and how the latent representations of language models might align with the human brain processing of multiple levels of content. in this study, we address these issues by recording human brain activity using functional magnetic resonance imaging (fmri) while participants view 8.3 hours of videos of dramas or movies. importantly, we heavily annotate these videos across multiple levels of semantic content related to the drama, including speech dialogue, visual objects, and background story. we extract latent representations of these annotations from llms, then build encoding models that predict brain activity from these latent representations to quantitatively compare how each type of information is represented in different brain regions. furthermore, we quantitatively assess how each brain region uniquely captures differ-ent aspects of semantic content using different types of latent representations derived from llms and multi-modal llms. our contributions are as follows: ## related work in numerous previous studies, researchers examined the relationship between the latent representations of language models and brain activity during speech comprehension. these researchers primarily focused on the correspondence between latent representations of language models obtained from speech transcriptions and brain activity @xcite @xcite @xcite . recent findings have further demonstrated that llms provide a better explanation of brain activity than traditional language models @xcite @xcite . regarding the semantic content of the visual object in the scene, the correspondence between the latent representations of deep learning models related to the displayed objects and human brain activity have been studied extensively @xcite @xcite @xcite @xcite @xcite @xcite . in most of these studies, researchers used neuroimaging data while participants watched static images or short video clips. although several neuroscience studies have been conducted in which researchers explored high-level story content in the brain using a naturalistic movie watching experiment @xcite @xcite @xcite , these researchers typically did not explicitly model high-level story content using language models. furthermore, they have not examined the unique explanatory power story-specific semantic content exerts on brain activity compared with other types of semantic content. there is a growing consensus that llms mirror human brain activity more accurately than traditional language models during semantic comprehension; however, in the individual studies described above, the reseachers addressed discrete aspects of semantic comprehension independently. this is problematic because humans process different types of semantic content simultaneously in naturalistic scenarios, and such content may be represented differently in llms and the human brain. here, we address these issues by evaluating how much each level of semantic content uniquely explains brain activity compared with other semantic content. ## discussion and conclusions in this study, we quantitatively compared the relationship between different latent representations of llms and brain activity using diverse annotations of semantic content. to achieve this, we collected 8.3 hours of an fmri dataset of brain activity recorded while participants watched extensively annotated videos. we demonstrated that the llms' latent representations explain brain activity particularly well for high-level background story content compared with traditional language models. moreover, our results show that different levels of semantic content are distributed differently in the brain. finally, we demonstrated that multi-modal vision-semantic models explain brain activity better and more uniquely than unimodal models. while our findings align with prior work, we extend it by simultaneously providing a more detailed analysis of uniquely explained variances across multiple levels of semantic representations and brain regions, particularly when comparing multimodal and unimodal models. overall, our findings provide valuable insights into interpreting llms from a neuro-ai perspective, offering a finer-grained understanding of the similarities between human cognition and ai compared to previous studies. we reemphasize that these insights were not apparent from the modeling of individual features, as was performed in previous studies. for instance, the absence of unique variance for summary and timeplace is a significant insight, which indicates that simply including these types of information in encoding analysis might not be sufficient for capturing high-level semantic representations in the brain. our results do not indicate that such information is absent in the brain but the absence could be caused by the limitations of the modeling approaches or fmri measurement. hence, in future work, we need to consider how to model these types of high-level information using our data as a new biological benchmark for alignment between llms and humans.