Investigating Rich Feature Sources for Conceptual Representation Encoding (2020)

Abstract: Functional Magnetic Resonance Imaging (fMRI) provides a means to investigate human conceptual representation in cognitive and neuroscience studies, where researchers predict the fMRI activations with elicited stimuli inputs. Previous work mainly uses a single source of features, particularly linguistic features, to predict fMRI activations. However, relatively little work has been done on investigating rich-source features for conceptual representation. In this paper, we systematically compare the linguistic, visual as well as auditory input features in conceptual representation, and further introduce associative conceptual features, which are obtained from Small World of Words game, to predict fMRI activations. Our experimental results show that those rich-source features can enhance performance in predicting the fMRI activations. Our analysis indicates that information from rich sources is present in the conceptual representation of human brains. In particular, the visual feature weights the most on conceptual representation, which is consistent with the recent cognitive science study.

## introduction how a simple concept is represented and organized by human brain has been of long research interest in cognitive science and natural language processing (nlp) @xcite @xcite . the rise of brain imaging methods such as fmri technology has now made it feasible to investigate conceptual representation within human brain. in particular, fmri is a technique that allows for the visualization of neuron activity in brain regions, which has become an essential tool for analyzing the neural correlates of brain activity in recent decades @xcite @xcite @xcite . neuroscientists have shown that distinct patterns of neural activation are associated with both encoding and decoding the concepts of different semantic categories in brains. @xcite first introduced the task of predicting fmri activation and proposed a featured-based model which takes a semantic representation of a single noun to predict the fmri activation elicited by that noun. subsequent studies @xcite introduced distributed based methods to build correlations between distributed semantic representations and patterns of neural activation. however, previous work mostly focuses on a single source of input features, e.g. count-based word vectors @xcite @xcite to explore the in brain encoding process, which builds correlation between neural signals and distributed representation, and thus can be useful for better understanding both the brain and the word representation. but there has been little work systematically investigating the effect of different modalities on predicting fmri activations. we address this limitation by empirically investigating two forms of rich source features: multimodal features and associative conceptual feature. first, we systematically compare input features that come from linguistic, visual and auditory sources into fmri activation encoding. to investigate the influence of each source of information in the brain conceptual representation, we build and evaluate a multimodal ## related work previous studies on conceptual representation mainly focus on correlation between words and corresponding fmri activations, including feature based methods and distributed representation based methods. seminal work of @xcite pioneered the use of corpus-derived word representations to predict brain activation data associated with the meaning of nouns. this feature based method selected 25 verbs (i.e., 'see', 'say', 'taste'.), and calculated the co-occurrence frequency of the noun with each of 25 verbs. in this regard, a noun word is encoded into 25 sensor-motor features. subsequent work including @xcite used wordnet @xcite to compute the values of the features. obviously, such feature based methods are constrained by corpora, and only focus on linguistic unimodal. @xcite proposed a distributed semantics based method using features learnt form wikipeida to predict neural activations for unseen concepts. since then, various studies have shown that distributed semantic representations have correlations with brain concept representation @xcite @xcite @xcite . however, though these methods outperform the feature based methods, they still ignore the fact that the information in the real world comes as different modalities. in contrast to their work, we investigate the human conceptual representation mechanism via evaluating the effects of multimodal features rather than only unimodal linguistic feature. more closely related to our work, @xcite presented a systematic evaluation and comparison of unimodal and multimodal semantic models in their ability to predict patterns of conceptual representation in the human brain. however, they only focused on the model level, contrasting unimodal representations and multimodal representations that involve linguistic and visual signals, but not the effect of each modality. while little previous work studied the influence of each source of information in the brain conceptual representation, our study is more extensive by evaluating multiple modalities data and their combinations. to our knowledge, we are the first to report auditory data in exploring human conceptual representations. more vitally, we explore their importance in concrete noun representations. different from all work above, we are also the first to introduce associative conceptual words as input features to human conceptual representation. ## task: predicting the fmri activation the task is to predict the corresponding fmri activations with elicited stimuli. the encoder operates by predicting fmri activation given feature vectors. each dimension (voxel) of fmri activation is predicted by using a separate ridge regression estimator. more formally, given the matrix x and the matrix z, we learn regression coefficients b and b 0 that minimize for each column of z of z matrix. x is the semantic matrix, the dimension is the number of words (training set) by the dimension of semantic vector (300 for glove); and z is the corresponding fmri activation matrix, the dimension is the number of fmri activation by the imaging dimension (amount of selected voxel, 500 for @xcite dataset and 5000 for pereira et al. (2018) dataset). we investigate three types of multi-sense inputs, namely, linguistic, visual and auditory sources. and further we use associative conceptual input, namely, the associative conceptual words which is obtained from small world of word game. in the next two sections, we will introduce how to obtain multi-sense representations and associative conceptual representations. ## multi-sense representations following @xcite and @xcite , we construct multimodal semantic representation vector, v m , by concatenating the linguistic, visual and auditory representations as shown in figure 1 : ## associative conceptual representation associative conceptual representation is a dense vector obtained from the associative conceptual words that are produced by humans in a game scene, and it is used to presented human's associative thinking related a concept. to investigate that whether associative thinking can be reflected in the fmri activation, we fuse the word vectors linearly and use it as our associative conceptual representations. the linear fusion is represented as: ## experiments we apply three sources of features to predict fmri activations with unimodal model and multimodal models, and compare their performances. further, we compare the performances of models with irrelevant words and associative conceptual words as inputs respectively. 6.1 datasets 6.1.1 fmri datasets in this paper, we use the fmri activation datasets of @xcite and @xcite . the summary of the datasets is shown in pereira et al. (2018)'s fmri activation dataset was collected from 16 subjects. similarly to @xcite , subjects were asked to think about the properties when they were presented with stimulus in form of words, pictures and sentences. but the exemplar words of @xcite cover a broader semantic vector space and are more distinct in vector space. first, they applied 300-dimensional glove @xcite to obtain semantic vectors for all words in a vocabulary size of approximately 30,000 words @xcite . they then utilized spectral clustering @xcite to group the vectors into 180 regions, and hand-selected 180 representative words for each regions. ## conclusion and future work we explored conceptual representation in human brains by evaluating the effect of multimodal data in predicting fmri activation, observing a clear advantage in predicting brain activation for visually grounded models. this finding consistent with the neurological evidence that the word comprehension first involves activation of shallow language-based conceptual representation , which is then complemented by deeper simulation of visual properties of the concept @xcite . from the associative thinking perspective, we find that though the associative words might be far away in the distributed semantic vector space, we could still use them to better predict fmri activation. we carried out more thorough and extensive work compare to the work of @xcite . the findings also support the hypotheses that the linguistic, conceptual and perceptual systems interplay in the human brain @xcite . the fmri datasets used in our study are generated by presenting subjects with written words together with pictures. in other words, the fmri representations are the participants' reactions to linguistic and visual input -but not acoustic. to further study human brain response representations to the acoustic stimuli, we plan to collect fmri when presenting acoustic concepts.