Neural Language Taskonomy: Which NLP Tasks are the most Predictive of f MRI Brain Activity? (2022)

Abstract: Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporo-parieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.

## introduction brain encoding aims at constructing neural brain activity given an input stimulus. since the discovery of the relationship between language stimuli and functions of brain networks using fmri [for ex., @xcite ], researchers have been interested in understanding how the neural encoding models predict the fmri brain activity. several brain encoding models have been developed to (i) understand the ventral stream in biological vision @xcite @xcite , and (ii) to study the higher-level cognition like language processing @xcite @xcite . some recent studies @xcite have been able to identify brain rois (region of interest) that respond to words that have a similar meaning and have thus built a "semantic atlas" of how the human brain organizes language. further, several studies @xcite @xcite have used a wide variety of word embeddings where words represented as vectors in an embedding space are mapped to brain activation for improved neural coding. recently, transformer @xcite ) based models like bert @xcite have been found to be very effective across a large number of natural language processing (nlp) tasks. these transformer based models have been pretrained on millions of text instances in an unsupervised manner and further finetuned to specialize for various nlp tasks. natural language understanding requires integrating several cognitive skills like syntactic parsing of the language structure, identifying the named entities, capturing the word meaning in the context, coreference resolution, etc. learning from massive corpora enables these models to excel at cognitive skills required for language understanding. interestingly, such transformer-based neural representations have been found to be very effective for brain encoding as well @xcite . despite the recent advances in mapping between language transformers and the brain activity recorded with reading @xcite , the transformer features themselves are notoriously difficult to interpret. in recent works, @xcite ; @xcite address this issue by disentangling the high-dimensional transformer representations of language models into four combinatorial classes: lexical, compositional, syntactic, and semantic representations to explore which class is highly associated with language cortical rois. representations do not exist in a vacuum but become meaningful only when they accomplish a task. therefore, the next logical step is to see which of these transformer representations most effectively drive the linear mapping between language models and the brain in the context of nlp tasks. @xcite fine-tune a pretrained bert model on multiple tasks to find tasks best correlated with high decoding performance. in this study, we investigate the correlation between brain activation and feature representations learned by different task-specific networks, and ask which tasks lead to improvements in brainencoding performance. recently, a study using multiple computer vision tasks has shown that 3d vision task models predict better fmri brain activity than 2d vision task models @xcite for visual stimuli. inspired by the success of correlations in the vision field @xcite , and brain encoding study of a variety of language transformer models @xcite , we build neural language taskonomy models for brain encoding and aim to find nlp tasks that are most explanatory of brain activations for reading and listening tasks. in this paper, we uncover insights about the association between fmri voxel activations and representations of diverse nlp tasks representations. the predictive power of task-specific representations with brain activation is ascertained by (1) using ridge regression on such representations and predicting activations and (2) computing popular metrics like 2v2 accuracy and pearson correlation between actual and predicted activations. specifically, we make the following contributions in this paper. ## related work older methods for text-based stimulus representation include text corpus co-occurrence counts @xcite @xcite , syntactic and discourse features @xcite . in recent times, both semantic and experiential attribute models have been explored for text-based stimuli. semantic representation models include distributed word embeddings @xcite @xcite @xcite , sentence representation models @xcite @xcite , recurrent neural networks @xcite , and transformer-based language models @xcite @xcite . experiential attribute models represent words in terms of human ratings of their degree of association with different attributes of experience, typically on a scale of 0-6 @xcite @xcite @xcite or binary @xcite . fine-grained details such as lexical, compositional, syntactic, and semantic representations of narratives are factorized from transformer-based models and utilized for training encoding models. the resulting models are better able to disentangle the corresponding brain responses in fmri @xcite . in this paper, we focus on transformer-based linguistic stimuli representations since they have been found to be most effective. unlike previous studies which directly used existing task-agnostic pretrained models, we train task-specific transformer models and aim to find which model leads to the best encoding accuracy given reading and listening language stimuli. ## brain imaging datasets we work with two datasets: pereira and narratives-pieman. results on narratives-lucy and narratives-slumlord show similar trends. hence, we also show results on narratives-lucy and narratives-slumlord in the appendix. pereira dataset (reading sentences from passages) for the pereira dataset, similar to earlier work @xcite , we combine the data from sentence-based experiments (experiments-2 and 3) from @xcite . five subjects were presented a total of 627 sentences from 48 broad topics, spanning over 168 passages, where each passage consists of 3-4 sentences. as in @xcite , we focused on nine brain rois (regions of interest) corresponding to four brain networks: (i) default mode network (dmn) (linked to the functionality of semantic processing), (ii) language network (related to language processing, understanding, word meaning, and sentence comprehension), (iii) task positive network (tp) (related to attention, salience information), and (iv) visual network (related to the processing of visual objects, object recognition). we briefly summarize the details of the dataset and the number of voxels corresponding to each roi in narratives-pieman (listening to stories) the "narratives" collection aggregates a variety of fmri datasets collected while human subjects listened to naturalistic spoken stories. the narratives dataset that includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ∼4.6 hours of unique stimuli (∼43,000 words) was proposed in @xcite . similar to earlier works @xcite , we analyze data from 82 subjects listening to the story titled 'pieman' with 259 trs (repetition time -fmri recorded every 1.5 sec.). we list number of voxels per roi in this dataset in ## encoding model to explore how and where contextual language features are represented in the brain when reading sentences and listening to stories, we extract different features spaces describing each stimulus sentence and use them in an encoding model to predict brain responses. our reasoning is as follows. if a feature is a good predictor of a specific brain region, information about that feature is likely encoded in that region. in this paper, for both datasets, we train fmri encoding models using ridge regression on stimuli representations obtained using a variety of nlp tasks. the main goal of each fmri encoder model is to predict brain responses associated with each brain region given a stimuli. in all cases, we train a model per subject separately. following literature on brain encoding @xcite , we choose to use a ridge regression model instead of more complicated models. we plan to explore more such models as part of future work. we follow k-fold (@xmath0. all the data samples from k-1 folds were used for training, and the model was tested on samples of the left-out fold. we used sklearn's ridge-regression with default parameters, 10-fold cross-validation, stochastic-average-gradient descent optimizer, huggingface for transformer models, mse loss function, and l2-decay (λ) as 1.0. we used bert word-piece tokenizer for the linguistic transformer input. all experiments were conducted on a machine with 1 nvidia geforce-gtx gpu with 16gb gpu ram. we make the code publicly available 1 . ## results in order to assess the performance of the fmri encoder models learned using the representations from a variety of nlp tasks, we computed the 2v2 accuracy and pearson correlation coefficient between the predicted and true responses across various rois for both the reading (pereira) dataset (fig. 1 ) as well as the listening (narratives-pieman) dataset (fig. 2 ).