Can Brain Signals Reveal Inner Alignment with Human Languages? (2023)

Abstract: Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream tasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency between EEG and language. To study at the representation level, we introduced MTAM , a M ultimodal T ransformer A lignment M odel, to observe coordinated representations between the two modalities. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 1.7% on K-EmoCon and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCo for relation detection. In addition, we provide interpretations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language; (2) alignment weights show the influence of different language semantics as well as EEG frequency features; (3) brain topographical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code is available at https://github.com/Jason-Qiu/EEG_Language_Alignment .

## introduction brain activity is an important parameter in furthering our knowledge of how human language is represented and interpreted @xcite @xcite @xcite . researchers from domains such as linguistics, psychology, cognitive science, and computer science * *marked as equal contribution have made large efforts in using brain-recording technologies to analyze cognitive activity during language-related tasks and observed that these technologies added value in terms of understanding language @xcite . basic linguistic rules seem to be effortlessly understood by humans in contrast to machinery. recent advances in natural language processing (nlp) models @xcite have enabled computers to maintain long and contextual information through self-attention mechanisms. this attention mechanism has been maneuvered to create robust language models but at the cost of tremendous amounts of data @xcite @xcite @xcite . although performance has significantly improved by using modern nlp models, they are still seen to be suboptimal compared to the human brain. in this study, we explore the relationship and dependencies of eeg and language. we apply eeg, a popularized routine in cognitive research, for its accessibility and practicality, along with language to discover connectivity. our contributions are summarized as follows: multimodal learning of language and other brain signals wehbe et al. (2014) used a recurrent neural network to perform word alignment between meg activity and the generated word embeddings. toneva and wehbe (2019) utilized wordlevel meg and fmri recordings to compare word embeddings from large language models. schwartz et al. (2019) used meg and fmri data to fine-tune a bert language model (devlin et al., 2019) and found the relationships between these two modalities were generalized across participants. huang et al. (2020) leveraged ct images and text from electronic health records to classify pulmonary embolism cases and observed that the multimodal model with late fusion achieved the best performance. however, the relationship between language and eeg has not been explored before. multimodal learning of eeg and language foster et al. (2021) applied eeg signals to pre-dict specific values of each dimension in a word vector through regression models. wang and ji (2021) used word-level eeg features to decode corresponding text tokens through an open vocabulary, sequence-to-sequence framework. hollenstein et al. (2021) focused on a multimodal approach by utilizing a combination of eeg, eye-tracking, and text data to improve nlp tasks, but did not explore the relationship between eeg and language. more related work can be found in appendix e. ## conclusion in this study, we explore the relationship between eeg and language. we propose mtam, a multimodal transformer alignment model, to observe coordinated representations between the two modalities and employ the transformed representations for downstream applications. our method achieved state-of-the-art performance on sentiment analysis and relation detection tasks on two public datasets, zuco and k-emocon. furthermore, we carried out a comprehensive study to analyze the connectivity and alignment between eeg and language. we observed that the transformed features show less randomness and sparsity. the word-level language-eeg alignment clearly demonstrated the importance of the explored connectivity. we also provided brain topologies as an intuitive understanding of the corresponding activity regions in the brain, which could build the empirical neuropsychological basis for understanding the relationship between eeg and language through computational models.