A Deep Autoencoder for Near-Perfect fMRI Encoding
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Encoding models of functional magnetic resonance imaging (fMRI) data attempt tolearn a forward mapping that relates stimuli to the corresponding brain activation.
Computational tractability usually forces current encoding as well as decodingsolutions to typically consider only a small subset of voxels from the actual 3Dvolume of activation.
Further, while brain decoding (reconstructing stimulusinformation from the brain activation) has received wider attention, there have beenonly a few attempts at constructing encoding solutions in the extant neuroimagingliterature.
In this paper, we present a deep autoencoder consisting of convolutionalneural networks in tandem with long short-term memory (CNN-LSTM) model.
Themodel is trained on fMRI slice sequences and predicts the entire brain volume ratherthan a small subset of voxels from the information in stimuli (text and image).
Weargue that the resulting solution avoids the problem of devising encoding modelsbased on a rule-based selection of informative voxels and the concomitant issueof wide spatial variability of such voxels across participants.
The perturbationexperiments indicate that the proposed deep encoder indeed learns to predict brainactivations with high spatial accuracy.
On challenging universal decoder imagingdatasets (Pereira et al., 2018), our model yielded encouraging results.
Introduction
Apart from clinical use for diagnosing a variety of clinical conditions such as depression
Alzheimer’sdementia etc. functional magnetic resonance imaging (fMRI) studies are conducted extensively inneuroscience research to understand how knowledge is represented in the brain.
Since the workof Mitchell et al.
(2008) there has been an increasing interest in using computational models tointerpret neural activity using either the decoding or encoding models (stimulus features are usedto model brain activity)
(Naselaris et al., 2011; Mesgarani et al., 2014; Di Liberto et al., 2015).
Anencoding model that predicts brain activity in response to stimuli is important for neuroscientistswho can use the model predictions to investigate and test hypotheses about the transformation fromstimulus to brain response in patients.
In the context of fMRI, the voxel response is a proxy for brainactivity and so a fMRI encoding model predicts voxel responses.
Recent approaches of modeling fMRI data use training data set to estimate a separate model foreach recorded voxel.
Together, these models describe how information of the sensory stimulus orvisual function is encoded in the measured brain activity (Naselaris et al., 2011).
Word embeddingrepresentations were used to build encoding systems (Oota et al., 2018;
Abnar et al., 2018).
Somemethods rely on the parametric regression that assumes that the response is linearly related to stimulusfeatures after ﬁxed parametric nonlinear transformation(s) (Mitchell et al., 2008).
However, it isvery difﬁcult to estimate a model with minimal training data, especially when there are hundreds ofstimulus features that need to be mapped to thousands of voxels.
In this paper, we present an autoencoding model that predicts the complete brain activity associatedwith multi-modal forms of concrete nouns, which include words and images.
The theory underlying this computational model is that when the autoencoder is trained on sufﬁciently large corpus, themodel can transform the stimulus S which is either a word or image (or both) into corresponding3D brain encoding E .
To meet the demand for larger training corpus for deep learning models, wesplit the 3D volume into several 2D slices.
We present experimental evidence showing that the bestencoding model is achieved when it is presented with multi-modal stimulus information rather thanwords or images alone.
fMRI Encoding: Our Approach
Traditional methods either used a set of selective voxels from the dataset (Anderson et al., 2017;
Pereira et al., 2018) or handpicked region-based voxels to model brain encoding (Oota et al.
, 2018)and decoding analysis.
In the next sections, we discuss the disadvantages of such methods and ourenhancements to overcome these issues.
Voxels and Semantic slices:
A voxel is a three-dimensional rectangular cuboid and smaller voxelscontain fewer neurons on average and hence have less signal than larger voxels.
The three-dimensionalvolume of the subject’s head comprises several voxels arranged sequentially and can be unfolded intoa single line (raster coding).
Earlier studies used a subset of voxels for learning encoding modelsusing multiple regression to obtain maximum likelihood estimates of the voxel values.
That is, obtaina set of voxel values that minimizes the sum of squared error in reconstructing the training fMRIimages (Mitchell et al., 2008; Jain & Huth, 2018).
Though earlier experiments were conducted with minimal subsets, behavioral and long-term studiesof subjects may require generation of the entire 3D volume when the subject is tested with variousstimuli (Nie et al., 2016).
This creates a necessity for encoding models that are capable of generatinga complete 3D volume of the subject’s brain based on past fMRI history.
We attempted to performthe task of predicting complete 3D volume by utilizing all voxels in the training data (Pereira et al.,2018), converting them to sequences of 2D slices.
We argue that the slices provide enough semanticencoding information to train a sequential spatial model, since we observed a gradual change inactivation in regions across multiple slides, as seen in Figure 1.
[Figure]
Architecture:
We used a CNN-LSTM based autoencoder model, whose architecture is inspiredfrom Vinyals et al. (2015).
Figure 2 describes a basic overview, where CNNs are used for fMRIslice encoding and decoding and LSTMs to learn temporal/semantic features across slides.
Boththe encoder and decoder have CNN layers with 64, 128 and 256 ﬁlters, respectively.
Two layersof LSTMs (256, 128) were used as latent layers.
The multi-modal features of text and image, passthrough two independent layers of LSTM before concatenating to the outputs of CNN encoder.
Themodel uses fMRI slice inputs and “one step ahead” slices as outputs during training.
During testing,only the multi-modal input (image, word embedding, and start slice) is given to initiate the cascadeof predictions.
The modal uses its own output at time step t as input in time step t+1 .
Multi-modal Semantic models:
In Multi-modal semantics (Bruni et al., 2014), a model takes acorpus of images with relevant word vectors as input and ﬁnds a correspondence between the twomodalities.
For the linguistic input, we use the popular context-predicting text-based semantic model
GloVe (Pennington et al., 2014) to obtain a 300-dimensional word embedding which represents theconcept word.
Image representation comprising 2048 features is obtained by using the output of thefully connected layer of pre-trained Xception (Simonyan & Zisserman, 2014) model.
We retrieved 5images per word from the image stimuli corpus for the 180 concepts (pictures) of the experiment to input to LSTM a slice ( input to the CNN model.
Experiments
Dataset:
We used data from paradigm 1 of fMRI experiment 1 (Pereira et al., 2018), where authorsconducted experiments with multiple subjects by showing various forms of stimulus (sentence,word+picture, or both).
Paradigm 1 contains three experiments.
(i)In the ﬁrst experiment, the targetword was presented in the context of a sentence that made the relevant meaning salient.
(ii)In thesecond, the target word was presented with a picture that depicted some aspect(s) of the relevantmeaning.
(iii)In the third, the target word was presented in a multi-modal form where both wordand image were used.
This fMRI dataset was collected from a total of 16 participants.
For eachparticipant in paradigm 1, a total set of 180 words (128 nouns, 22 verbs, 29 adjectives and adverbs,and 1 function word) were used as stimuli in multi-modal form (word, picture).
The dataset containsfMRI captured as 128 × 88 voxel windows arranged as 85 slices, per subject per stimulus.
Out of 85slices, we ignored the initial 9 slices and the last 7 slices due to no activation present in any of thebrain regions.
and Discussion:
Using the approach discussed in Section 2, we trained separate encodingmodels per experiment for each subject.
The encoding performance was evaluated by training andtesting models using different subsets of the 180 concepts in a 5-fold cross-validation scheme.
[Table]
The encoder models were trained until the epochsstopped due to early stopping method, when validationloss did not change for few epochs.
We observed anaverage validation loss of 0.0007 for word based models and 0.0003 validation loss for multi-modal modelacross all tested subjects.
In order to assess the similarity between the actual and predicted brain slice, wecompared the slice-wise voxel coordinates and intensityof the voxels.
We measured the precision, recall, and F1-scores using voxel intensities and location of voxelcoordinates between the predicted and actual slice data.
Table 1 depicts the performance comparison betweentext alone model versus the model trained on multimodal stimulus information.
Although the precision,recall, F1-scores of two modalities are nearly similar,from Figure 1, we observe that the similarities betweenground truth and cortical brain responses from multi-modal based encoding model are better.
Some ofthe voxel intensity values predicted by the GloVe embedding model are very negligible in certain brainregions, which cause no activation.
Figure 3 shows the similarity (correlation) matrix between actualand predicted brain response with multi-modal stimuli and word embedding stimulus.
The correlationmatrix is calculated by considering both the actual and predicted voxels in every brain slice.
Weconsidered voxels with high activations, that is, those with intensity values greater than a threshold(= mean + standard deviation) and discarded the remaining voxels with low activation values.
Here,we found reliable correlations between fMRI responses from trained model and the actual brainresponses for all the test words in the case of the model trained with multi-modal information ascompared to word embedding information alone.
Perturbation experiments (results not shown here)where the random input is given as stimulus to the trained model yielded brain responses that hadminimal correlation with any of the semantic encodings for the 180 concepts.
These results verify therobustness of the learned encoding model.
Conclusion
In this work, we proposed an encoder model which can generate a complete 3D model of the brainusing multi-modal input, by training the model on subject’s brain response for words in the trainingset.
Different from previous work, our method predicts the complete set of voxels, as given in thedataset rather than selected few voxels per subject.
The key distinction of our work is the utilizationof machine translation inspired encoder-decoder model to generate complete brain image.
In thefuture, we plan to experiment on all paradigms and experiments mentioned in the dataset, with aprimary focus on attention-based autoencoder.