Visio-Linguistic Brain Encoding (2022)

Abstract: Brain encoding aims at reconstructing fMRI brain activity given a stimulus. There exists a plethora of neural encoding models which study brain encoding for single mode stimuli: visual (pretrained CNNs) or text (pretrained language models). Few recent papers have also obtained separate visual and text representation models and performed late-fusion using simple heuristics. However, previous work has failed to explore the co-attentive multi-modal modeling for visual and text reasoning. In this paper, we systematically explore the efficacy of image and multi-modal Transformers for brain encoding. Extensive experiments on two popular datasets, BOLD5000 and Pereira, provide the following insights. (1) We find that VisualBERT, a multi-modal Transformer, significantly outperforms previously proposed single-mode CNNs, image Transformers as well as other previously proposed multi-modal models, thereby establishing new state-of-the-art. (2) The regions such as LPTG, LMTG, LIFG, and STS which have dual functionalities for language and vision, have higher correlation with multi-modal models which reinforces the fact that these models are good at mimicing the human brain behavior. (3) The supremacy of visio-linguistic models raises the question of whether the responses elicited in the visual regions are affected implicitly by linguistic processing even when passively viewing images. Future fMRI tasks can verify this computational insight in an appropriate experimental setting. We make our code publicly available.

## introduction brain encoding aims at constructing neural brain activity recordings given an input stimulus. the two most studied forms of stimuli include vision and language. since discovering of the relationship between language/visual stimuli and functions 1 https://tinyurl.com/vlbencoding of brain networks @xcite , researchers have been interested in understanding how the neural encoding models predict the fmri (functional magnetic resonance imaging) brain activity. recently, several brain encoding models were developed to (i) understand the ventral stream in biological vision @xcite @xcite and (ii) study higher-level cognition like language processing @xcite @xcite . previous work has mainly focused on independently understanding vision and text stimuli. however, the biological systems perceive the world by simultaneously processing high-dimensional inputs from diverse modalities such as vision, auditory, touch, and proprioception @xcite . in particular, how the brain effectively processes and provides its visual understanding through natural language and vice versa is still an open question in neuroscience. earlier studies mainly were related to neural encoding models that predict brain activity using representations of single-mode stimuli: visual or text. convolutional neural networks (cnns) were known to encode semantics from visual stimuli effectively. interestingly, intermediate layers in deep cnns trained on the imagenet @xcite categorization task can partially account for how neurons in intermediate layers of the visual system respond to any given image @xcite @xcite @xcite . however, more recent and deeper cnns did not further improve on measures of brain-likeness, even though their imagenet performance has vastly increased @xcite . recently, @xcite proposed a shallow recurrent anatomical network, cornet, which provided state-of-theart results on the brain-score @xcite benchmark. similar to cnn based visual encoding models, various studies leveraged sa ffn image transformer sa ffn transformer layers sa ffn 2v2 accuracy / pearson correlation predicted actual ridge regression l12 l6 l1 faster r-cnn region features [cls] a tennis player swinging his racket on a court visualbert lxmert transformer (12 layers) cross-modality transformer object relationship transformer (9 layers) language transformer (12 layers) ridge regression ridge regression faster r-cnn region features ## brain imaging datasets the following datasets are popularly used in the literature for studying brain encoding: vim-1 @xcite , harry potter @xcite , bold5000 @xcite , algonauts @xcite , and ss-fmri @xcite . vim-1 has only black and white images, is only related to object recognition, and is subsumed by bold5000. ss-fmri is smaller and very similar to bold5000. the harry potter dataset does not have images. lastly, fmris have not been made publicly available for the algonauts dataset. hence, we experiment with bold5000 and @xcite datasets in this work. bold5000: bold5000 dataset was collected from four subjects where three subjects viewed 5254 natural images (imagenet: 2051, coco: 2135, scenes: 1068) while fmris were acquired. the fourth subject was shown 3108 images only. details of the visual stimuli and fmri protocols of the dataset have been discussed in @xcite . we briefly summarize the details of the dataset in we show number of instances and voxel distribution across various brain regions for the bold5000 and pereira datasets in tables 1 and 2 respectively. ## task descriptions we train fmri encoding models using ridge regression on stimuli representations obtained using various models for both datasets, as shown in fig. 1 . the main goal of each fmri encoder model is to predict fmri voxel values for each brain region given stimuli. in all cases, we train a model per subject separately. different brain regions are involved in processing stimuli involving objects and scenes. similarly, some regions specialize in understand-ing vision inputs while others interpret linguistic stimuli better. to evaluate the generalizability of our models across objects vs. scenes understanding, we also perform cross-data experiments where the train images belong to one sub-dataset, and the test images belong to the other sub-dataset. thus, for each subject, we perform (1) three same-sub-dataset traintest experiments and (2) six cross-sub-dataset traintest experiments. full dataset fmri encoding: whenever we train and test on the same dataset, we follow k-fold (@xmath0. all the data samples from k-1 folds were used for training, and the model was tested on samples of the left-out fold. cross-data fmri encoding: in the bold5000 dataset, we have three sub-datasets: coco, im-agenet, and scenes. imagenet images mainly contain objects. scenes images are about natural scenes, while coco images relate to both objects and scenes. for each of the three sub-datasets, we perform k-fold (@xmath1. ## methodology we trained a ridge regression-based encoding model to predict the fmri brain activity associated with the stimuli representation for each brain region. each voxel value is predicted using a separate ridge regression model. formally, we encode the stimuli as @xmath2, where n denotes the number of training examples, d denotes the dimension of input stimuli representation, and v denotes the number of voxels in a particular region. although ridge regression is a very naïve way of modeling, it has been the most popular brain encoding technique in this line of work. we plan to experiment with other forms of regression methods in the future. the input stimuli representation can be obtained using any of the following models: (i) pretrained cnns, (ii) pretrained text transformers (iii) image transformers, (iv) late-fusion models, or (v) multi-modal transformers. the ridge regression objective function for the i th example is given as follows. here, w are the learnable weight parameters, ∥.∥ f denotes the frobenius norm, and λ > 0 is a tunable hyper-parameter representing the regularization weight. λ was tuned on a small disjoint valida-tion set obtained from the training. next, we discuss different input stimuli representation methods. pretrained cnns and image transformers encode image stimuli only, while pretrained text transformers encode text stimuli only. late fusion models and multi-modal transformers encode both text and image stimuli. pretrained cnns: inspired by the algonauts challenge (cichy et al., 2019), we extract the layer-wise features from different pretrained cnn models such as vg-gnet19 @xcite (block1, block2, block3, block4, fc), in-ceptionv2resnet @xcite (conv2d5, conv2d50, conv2d100, conv2d150, conv2d200, conv2d_7b), and efficientnetb5 (tan and le, 2019) (conv2d2, conv2d8, conv2d16, conv2d24, fc), and use them for predicting fmri brain activity. we use adaptive average pooling on each layer to get features for each image. pretrained text transformers: roberta @xcite builds on bert's language masking strategy and has been shown to outperform several other text models on the popular glue nlp benchmark. we use the average-pooled representation foot_0 from roberta to encode text stimuli. image transformers: we used three image transformers: vision transformer (vit), data efficient image transformer (deit), and bidirectional encoder representation from image transformer (beit). given an image, image transformers output two representations: pooled and patches. we experiment with both representations. late-fusion models: in these models, the stimuli representation is obtained as a concatenation of image stimuli encoding obtained from pretrained cnns and text stimuli encoding obtained from pretrained text transformers. thus, we experiment with these late-fusion models: vggnet19+roberta, resnet50+roberta, inceptionv2resnet+roberta and efficient-netb5+roberta. these models do not incorporate real information fusion but do concatenation across modalities. multi-modal transformers: we experiment with these multi-modal transformer models: contrastive language-image pre-training (clip), learning cross-modality encoder representations from transformers (lxmert), and visualbert. these transformers take both image and text stimuli as input and output a joint visio-linguistic representation. specifically, the image input for these models comprises region proposals as well as bounding box regression features extracted from faster r-cnn (ren et al., 2015) as input features. these models incorporate information fusion across modalities at different levels of processing using co-attention and hence are expected to result in high-quality visio-linguistic representations. hyper-parameter settings: we used sklearn's ridge-regression with default parameters, 10-fold cross-validation, stochastic-average-gradient descent optimizer, huggingface for transformer models, mse loss function, and l2-decay (λ) as 1.0. we used word-piece tokenizer for the linguistic transformer input and faster-rcnn (ren et al., 2015) for extracting region proposals. all experiments were conducted on a machine with 1 nvidia geforce-gtx gpu with 16gb gpu ram. we make our code publicly available 1 . ## cognitive insights: does language influence vision? bold5000 dataset comprises brain responses from visual areas (early visual, scene-related, and objectrelated) when visual stimuli are presented to the subjects. although only visual information is present in the stimuli, it is conceivable that participants implicitly invoke appropriate linguistic representations that, in turn, influence visual processing @xcite . thus, it is not surprising that computational models such as multi-modal transformers (visualbert, and lxmert) that learn a joint representation of language and vision show superior performance on the 'purely' visual response data in bold5000 (see figs. 2 and 4(a) ). further, the performance of these models is naturally good in the case when text and image are shown to the participants, and whole-brain responses are captured as in the case of the pereira dataset (see figs. 3 and 4(b) ). we further investigate the role of different sub-rois of the language and visual networks. for this, we compare the predicted responses of the best encoding model, i.e., visualbert, with the ground truth (observed) responses of various language and visual sub-regions (see fig 8 ). we notice that the classical language areas in the temporal gyrus (lmtg and lptg) and the inferior frontal gyrus (lifg) are more accurately predicted than the other sub rois of the language network. these sub-rois (lmtg, lptg, and lifg) are highly involved in language comprehension and semantic processing. interestingly, the second-best correlations are seen for multi-modal integration areas in the temporo-parietal regions (langg, lfus, lpar) and higher-order processing and attention-related areas in the middle frontal region (mfg). in the visual sub-rois (fig. 8 ), we observe that the superior temporal sulcus (bilaterally but more in the left: lsts) is more accurately predicted than other sub-rois. surprisingly, lsts is implicated in various social processes, ranging from language perception to simulating the mental processes of others. also, the sub-rois such as lloc, lffa, lofa, and leba have a higher correlation. these areas are involved in more visual-related functions such as object recognition, face perception, face recognition, and body recognition. based on the intuition from the computational experiments, we make the following testable prediction for future fmri experiments. instead of a passive viewing task, if participants were to perform a naming task/decision-making task on the objects/scenes, we expect to see more pronounced and focused activation in the visual areas during the language-based task compared to passive viewing. ## conclusion we studied the effectiveness of multi-modal modeling for brain encoding. we found that visual-bert, which jointly encodes text and visual input using cross-modal attention at multiple levels, performs the best. our experiments on bold5000 and pereira datasets lead to interesting cognitive insights. these insights indicate that fmris reveal reliable responses in scenes and object selection visual brain areas, which shows that cross-view decoding tasks like image captioning or image tagging are practically possible with reasonable accuracy. we plan to explore this as part of future work. we also plan to explore correlations between brain voxel space and representational feature space in the future. finally, the combined strength of joint (audio, vision, and text) modalities remains to be investigated.