Multi-view and Cross-view Brain Decoding (2022)

Abstract: Can we build multi-view decoders that can decode concepts from brain recordings corresponding to any view (picture, sentence, word cloud) of stimuli? Can we build a system that can use brain recordings to automatically describe what a subject is watching using keywords or sentences? How about a system that can automatically extract important keywords from sentences that a subject is reading? Previous brain decoding efforts have focused only on single view analysis and hence cannot help us build such systems. As a first step toward building such systems, inspired by Natural Language Processing literature on multi-lingual and cross-lingual modeling, we propose two novel brain decoding setups: (1) multi-view decoding (MVD) and (2) cross-view decoding (CVD). In MVD, the goal is to build an MV decoder that can take brain recordings for any view as input and predict the concept. In CVD, the goal is to train a model which takes brain recordings for one view as input and decodes a semantic vector representation of another view. Specifically, we study practically useful CVD tasks like image captioning, image tagging, keyword extraction, and sentence formation. Our extensive experiments lead to MVD models with ~0.68 average pairwise accuracy across view pairs, and also CVD models with ~0.8 average pairwise accuracy across tasks. Analysis of the contribution of different brain networks reveals exciting cognitive insights: (1) Models trained on picture or sentence view of stimuli are better MV decoders than a model trained on word cloud view. (2) Our extensive analysis across 9 broad regions, 11 language sub-regions and 16 visual sub-regions of the brain help us localize, for the first time, the parts of the brain involved in cross-view tasks like image captioning, image tagging, sentence formation and keyword extraction. We make the code publicly available.

## introduction brain decoding models aim to understand what a subject is thinking, seeing, and perceiving by analyzing neural recordings. thus, in the context of language, it may be beneficial to learn mappings between linguistic representation and the associated brain activation, and how we compose the linguistic meaning from different stimuli such as text @xcite , images @xcite , videos @xcite , or speech @xcite by analyzing the evoked brain activity. also, decoding the functional activity of the brain has numerous applications in education and healthcare. brain recordings can be obtained by providing stimuli to a subject in various forms. for example, a concept (like apartment) can be presented using: (1) word picture (wp) view: picture along with the concept word, (2) sentence (s) view: sentence containing the word, or (3) word cloud (wc) view: word cloud containing the word along with other semantically related words. recent studies have made much progress using functional magnetic resonance imaging (fmri) brain activity to reconstruct semantic vectors corresponding to linguistic items, including words @xcite , phrases, sentences, and paragraphs @xcite . however, all such studies have been limited to single-view analysis. separate models are trained to process different views. also, the decoding target is typically a semantic vector of the concept word. in the natural language processing (nlp) community, researchers have recently started focusing on building multi-lingual and cross-lingual systems @xcite @xcite . multi-lingual systems 106 test the dishwasher can wash all the dishes. ## related work advances in functional neuroimaging tools such as fmri have made it easier to study the relationship between language/visual stimuli and functions of brain networks @xcite @xcite . a colorful bird sitting on a tree branch. a flock of red birds resting in their nest. a small red bird sitting on a snowcovered ground. initial brain decoding experiments studied the recovery of simple concrete nouns and verbs from fmri brain activity @xcite @xcite where the subject watches either a picture or a word. unlike the earlier work, @xcite built a model to decode the text passages instead of individual words. however, these studies used either simple or constrained sets of stimuli, which poses a question of generalization of these models. recently, @xcite explicitly decoded both words and sentences when subjects were shown both concrete and abstract stimuli. @xcite reconstructed the sentences along with categorizing words or predicting the semantic vector representation from fmri brain activity. @xcite ; @xcite focused on understanding how multiple tasks activate associated regions in the brain. to train ridge regression decoder models, earlier works focused on hand-crafted features @xcite , which suffer from various drawbacks like inability to capture the context and sequential aspects of a sentence, inability to extract signals for abstract stimuli, etc. with the success of deep learning based word representations, multiple researchers have used distributed word embeddings for brain decoding models in place of carefully hand-crafted feature vectors @xcite @xcite @xcite @xcite . using the distributed sentence representations, @xcite al. (2014b); jain and huth (2018); abnar et al. (2019); sun et al. (2019) demonstrated that neural sentence representations are better for decoding whole sentences from brain activity patterns. recently, transformer models like bert @xcite and gpt2 @xcite have been found to be very effective for decoding @xcite @xcite . inspired by such studies, we leverage bert representations. inspired by such studies, we leverage bert representations. unlike single-view analysis done in previous studies, multi-view and cross-view setups are the main focus of our work. ## results and cognitive insights since we are the first to propose multi-view and cross-view tasks, unfortunately, there are no baselines to compare with. for the sake of comparison, we design a "chance-level" bert (random) baseline where models are trained using bert embeddings of randomly chosen words as a target rather than bert embeddings of the actual target word. for same-view experiments, our results are in line with that reported in @xcite . ## evaluation metrics we use the popular pairwise and rank accuracy metrics for evaluation. pairwise accuracy to measure the pairwise accuracy, the first step is to predict all the test stimulus vector representations using a trained decoder model. let @xmath0, s 1 ,• • • ,s n ], ŝ = [ ŝ0 , ŝ1 ,• • • , ŝn ] denote the "true" (text-derived) and predicted stimulus representations for n test instances resp. given a pair (i, j) such that here, corr denotes the pearson correlation. final pairwise matching accuracy per participant is the average of scores across all pairs of test instances. ## conclusion we studied brain decoding in the context of zeroshot multi-view concept decoding and cross-view decoding tasks. we studied four cross-view decoding tasks: image captioning, image tagging, sentence formation, and keyword extraction. we show that cross-view decoding is feasible with good accuracy. brain network distribution analysis reveals insights about the importance of various parts of the brain for each of these tasks.