H ALLUCINATING BRAINS WITH ARTIFICIAL BRAINS
Anonymous authors
Paper under double-blind review
A BSTRACT
Human brain function as measured by functional magnetic resonance imaging ( fMRI)
, exhibits a rich diversity
In response, understanding the individual variability of brain function and its association with behavior has become one of the major concerns in modern cognitive neuroscience.
Our work is motivated by the view that generative models provide a useful tool for understanding this variability.
To this end this manuscript presents two novel generative models trained on real neuroimaging data which synthesize task-dependent functional brain images.
Brain images are high dimensional tensors which exhibit structured spatial correlations.
Thus, both models are 3D conditional Generative Adversarial networks (GANs) which apply Convolutional Neural Networks (CNNs) to learn an abstraction of brain image representations.
Our results show that the generated brain images are diverse, yet task dependent.
In addition to qualitative evaluation, we utilize the generated synthetic brain volumes as additional training data to improve downstream fMRI classiﬁers (also known as decoding, or brain reading).
Our approach achieves signiﬁcant improvements for a variety of datasets, classiﬁcation tasks and evaluation scores.
Our classiﬁcation results provide a quantitative evaluation of the quality of the generated images, and also serve as an additional contribution of this manuscript
I NTRODUCTION
Functional Magnetic Resonance Imaging (fMRI) is a common tool used by cognitive neuroscientists to investigate the properties of brain function in response to stimuli.
Classic analysis approaches (Poldrack et al., 2011) focused on analyzing group-averaged brain function images.
However, it was discovered that brain activation patterns vary signiﬁcantly between individuals.
Thus, modern analysis now prioritizes understanding the inter-subject variability of brain function (Dubois & Adolphs, 2016; Geerligs et al., 2017).
Our work is motivated by the view that generative models provide a useful tool for understanding this variability – as they enable the synthesis of a variety of plausible brain images representing different hypothesized individuals, and high-quality generative models can be analyzed to posit potential mechanisms that explain this variability (Horn et al., 2008).
The results presented in this paper provide – to our knowledge for the ﬁrst time, positive results suggesting that it is indeed possible to generate high quality diverse and task dependent brain images.
While we can qualitatively evaluate generative brain images, quantitative evaluation allows us to objectively compare between various results.
To this end, we utilize the generated synthetic brain volumes as additional training data to improve downstream fMRI classiﬁers.
The use of classiﬁers to predict behavior associated with brain images is also known as decoding or brain reading (Varoquaux & Thirion, 2014; Pereira et al., 2009).
Classiﬁers such as support vector machines and deep networks have been applied for decoding brain images.
For example, Cox & Savoy (2003) attempted to classify which of 10 categories of object a subject was looking at (including similar categories, such as horses and cows) based on limited number of brain images.
Besides visual tasks, Mitchell et al.
(2008) distinguished active regions of brains when subjects listened to linguistic words, where the stimuli included ﬁve items from each of 12 semantic categories (animals, body parts etc.).
Beyond providing a model for individual variability, high quality brain image synthesis addresses pressing data issues in cognitive neuroscience.
Progress in the neurosciences is stiﬂed by the difﬁculty of obtaining brain data either because of a limited culture of data sharing, or due to medical privacy regulations (Poldrack & Gorgolewski, 2014).
For the computational neuroscientist, gener- ated images deliver unlimited quantities of high quality brain imaging data that can be used to develop state of the art tools before application to real subjects and/or patients (Varoquaux & Thirion, 2014).
This approach of using modern generative models to synthesize data, which in turn accelerates scientiﬁc study, has already proven useful in many scientiﬁc ﬁelds such as particle physics and astronomy (Castelvecchi et al., 2017).
Our work represent a ﬁrst application or this approach to neuroscience.
One of the promising generative models are Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), capturing complex distributions using a non-cooperative two-player game formulation: a generator produces synthetic data by transforming samples drawn from a simple distribution; a discriminator focuses on distinguishing synthetic and real data.
Despite (or possibly due to) the compelling formulation, GAN training is known to be unstable.
To address this difﬁculty various variants have been proposed.
Wasserstein GANs (WGANs)
(Arjovsky et al., 2017) formulate the objective using the Wasserstein distance rather than the classical Jenson-Shannon divergence.
Improved training of WGANs (Gulrajani et al., 2017) applies an additional gradient penalty, which avoids the critical weight clipping in WGANs which might lead to pathological behavior.
Dualing
GANs restrict the discriminator and formulate the dual objective (Li et al., 2017).
Beyond improving the stability, conditional GANs (Mirza & Osindero, 2014) make it possible to control the data generation process by conditioning the model on additional information.
Auxiliary Classiﬁer GANs (AC-GANs) (Odena et al., 2016) unify a GAN and classiﬁer to a single architecture, employing labels to generate ImageNet samples.
3D GANs (Wu et al., 2016) reconstruct 3D objects and Edward & David (2017) propose to use improved WGAN to enhance the stability of 3D GANs.
We make the following contributions in this paper:
We develop Improved Conditional Wasserstein GANs (ICW-GAN) and Auxiliary Classiﬁer and Discriminator GANs (ACD-GAN), two types of 3D conditional GANs to synthesize fMRI brain data, both of which we ﬁnd to be stable to train.
We assess the qualitative quality and diversity of generated brain volumes.
Our results suggest that the proposed models are able to generate high quality task-dependent and diverse 3D brain images.
We evaluate our models on three datasets using a series of image classiﬁcation tasks with support vector machines and deep network classiﬁers at two levels of brain image resolution.
Results show that augmenting training data using synthetic data generated by our models can greatly improve test classiﬁcation accuracy of brain volumes.
R ELATED W ORK 2. N EURAL N ETS FOR F MRI B RAIN D ATA
Neural networks have been used by several authors for classifying brain imaging data.
Firat et al.
(2014) and Koyamada et al.
(2015), used 2D deep nets to extract features of fMRI brain images to classify brain states.
Nathawani et al.
(2016) applied both 2D and 3D neural networks to classify fMRI brain data.
Svanera et al.
(2017) decoded fMRI data of video stimuli and classify data into visual categories.
Similarly, Nathawani et al.
(2016) extracted features from 4-D fMRI data and used deep learning methods for discrimination of cognitive processes.
For manuscripts that apply both neural networks and other classiﬁers such as support vector machines, the classiﬁcation results with CNN are generally much better than using SVM in the single subject prediction task, which inspires us to continue to use neural networks in multiple subject classiﬁcation and to explore ways to improve it.
2. G ENERATIVE A DVERSARIAL N ETS
To learn a distribution over data x , a GAN (Goodfellow et al., 2014) formulates a 2-player noncooperative game between two deep nets.
The generator G uses a random noise vector z sampled from a prior distribution P z ( z ) as input and produces an image G ( z ) .
The generator is trained to fool the discriminator, which receives either synthetic data or real data and differentiates between them.
Formally, G and D play the following two-player minimax game with value function V ( G, D ) :
[Equation] [Equation]
For the conditional GAN (Mirza & Osindero, 2014), both G and D are conditioned on some extra information y , for instance, class labels or other features.
The conditioning can be presented by feeding y into both the generator and
The Wasserstein GAN (WGAN) (Arjovsky et al., 2017) uses Wasserstein-1 distance W(q,p) as the objective function.
It computes the minimum cost of transporting mass to transform the distribution q into the distribution p.
Under this circumstance, W(q,p) is continuous everywhere and its gradient with respect to its input was found to be more stable than its classical GAN counterpart.
Improved
Wasserstein GAN (IWGAN) (Gulrajani et al., 2017) argues that weight clipping of the critic, i.e., the discriminator, in WGANs inevitably causes the gradient to either vanish or to explode.
To address this issue Gulrajani et al.
(2017) propose an alternative penalty term in the critic loss based on the gradient norm.
In an auxiliary classiﬁer GAN (AC-GAN) (Odena et al., 2016), every generated sample has a corresponding label which can be used as a condition in the generator and the classiﬁer .
The discriminator is modiﬁed to contain an auxiliary decoder network to reconstruct the training data class.
Further, 3D-GANs (Wu et al., 2016) extend GANs to 3D object generation.
Different from classical
GANs, 3D-GANs apply the three dimensional convolution in both G and the D. By learning deep object representations, 3D GANs can generate visually appealing yet variable 3D object volumes.
A PPROACH
In the following we introduce two models for fMRI data generation: ICW-GAN and ACD-GAN.
They differ in model structure and in the way label information is taken into account.
We ﬁrst show the model structure of 3D ICW-GAN and its downstream classiﬁers in Section 3.1, and present
ACD-GAN in Section 3.2.
3. 3D I MPROVED C ONDITIONAL W ASSERSTEIN GAN (ICW-GAN)
Similar to classical generative adversarial networks (GANs), ICW-GANs are formulated as a noncooperative two-player game between two adversaries: (1) a generator ˆ x =
G θ ( z ) , which generates artiﬁcial samples ˆ x from randomly drawn latent encodings z via a transformation using a deep net parameterized by θ ; and (2) a discriminator D w ( x ) =
f w ( x ) which computes the probability that a given data point x is real rather than artiﬁcial, by using the logit f w obtained from a deep net parameterized by w .
Following Arjovsky et al.
, we build our models based on minimizing the Wasserstein distance via
[Equation]
where { f w } w ∈ W denotes a set of functions that are K-Lipschitz for some K. IW-GANs (Gulrajani et al., 2017) provide a more stable method for enforcing the Lipschitz constraint by employing a gradient penalty rather than weight clipping, when optimizing:
[Equation]
Hereby ˆ x is a convex combination of real data and artiﬁcal samples, i.e., ˆ x ← (cid:15) x
+ (1 − (cid:15) )
G ( z ) with (cid:15) ∼ U [0 , 1] .
In our models, we extend IW-GAN in two directions.
First, because fMRI data is three dimensional, 3D convolution and deconvolution are used in order to maintain the spatial structure and voxel information of the fMRI data.
Second, we condition both the discriminator D w and the generator
G θ on available labeled data.
As shown in Figure 1, our generator consists of three fully convolutional layers.
In convolutional layers, we use kernels with a size 4 × 4 × 4 and strides 2, with batch normalization and Leaky ReLU layers added between and a Tanh layer at the end.
The D looks like a mirror of the G but no activity function is used at the end of the D.
This ICW-GAN model follows models proposed in (Gulrajani et al., 2017; Mirza & Osindero, 2014;
Wu et al., 2016), however, we found the following modiﬁcations to stabilize training and supervise the generation process for classiﬁcation data: compared to existing GANs, our ICW-GAN applies a more stable upper bound for Wasserstein distance as an objective function, leverages conditioning labels to supervise the generation process, and utilizes 3D convolution methods for 3D object generation.
To include label information, we concatenate labels to the input and to the hidden layers.
At the input of the generator, one-hot labels are combined with the brain vector.
Then, for each of the intermediate layers, we use a fully connected layer followed by a tanh activation to transform the one-hot vector to a volume of appropriate size, i.e., 15 × 2 × 2 for the ﬁrst hidden layer, and 15 × 3 × 3 for the next.
We empirically found dimensions other than 15 , e.g., 3 to work well too.
We concatenate the label volume to intermediate volumes on feature dimension and pass the joint one to the next deconvolution layer.
We follow the same procedure in the architecture of the discriminator.
Referring to the volume labels via y , the objective function L of our ICW-GAN model is as follows:
[Equation]
where ˆ x ← (cid:15) x
+ (1 − (cid:15) )
G ( z ) , (cid:15) ∼ U [0 , 1] .
λ is a gradient penalty coefﬁcient.
We optimize both the discriminator and generator loss using Adam optimizer(Kingma & Ba (2014)).
D OWNSTREAM C LASSIFIERS FOR ICW-GAN
Our work is also motivated by the difﬁculty of acquiring fMRI data, so we consider the task of enlarging the dataset for downstream tasks such as classiﬁcation, using synthetically generated samples.
To assess the suitability of the developed generation procedure, we employ a variety of downstream classiﬁcation processes.
More speciﬁcally, we train a classiﬁer with real (‘Real’) or real plus generated (‘Real+G’) data.
Common classiﬁers in neuroscience are Support Vector Machines (SVMs) and a 3D deep networks.
We think usage of both provides valuable insights as their number of trainable parameters differ signiﬁcantly, i.e., 14 parameters for SVMs and several hundred thousand for our deep networks.
To measure the results obtained from different training datasets we use accuracy, macro F1, precision and recall metrics.
SVM : We use a simple linear SVM to classify test data and don’t extract any intermediate features.
Instead, we use raw brain data, vectorized to a 1-dimensional vector for classiﬁcation to avoid any form of collusion.
Scaling
Input
[Table]
Deep Net : The deep net structure is similar to the discriminator, having a 3 dimensional structure and identical number of convolution layers with a Leaky ReLU activation.
Obviously the classiﬁer doesn’t concatenate intermediate and input data with any label information.
We provide the respective information for each dataset in the experimental section.
3. 3D A UXILIARY C LASSIFIER AND D ISCRIMINATOR GAN (ACD-
GAN)
Instead of assessing the classiﬁer performance in a separate downstream classiﬁcation task, we also develop a conditional architecture of GANs which jointly trains the generator, the discriminator and the classiﬁer.
This form of training is inspired by the AC-GAN (Odena et al., 2016) formulation.
Our generator architecture is illustrated in Figure 1 and identical to the one employed for the ICW-GAN.
The combination of classiﬁer and discriminator is provided in Figure 2.
As before, the discriminator assesses the probability that a provided sample is real or artiﬁcially generated.
In addition, the classiﬁer provides a probability distribution over the class labels.
In this model, training passes labels through the generator and the discriminator in order to condition the adversarial process.
However, no labels are employed in the classiﬁcation stream.
The classiﬁer loss, L C , is the loglikelihood of the correct class.
Compared to L in Eq.
(4), the discriminator loss, L D includes the additional classiﬁer component
[Equation]
The generator loss function, L G , consists of the likelihood of fake images and one part of L C , i.e.,
[Equation]
Similarly, the discriminator loss function is given by L D =
L + L C .
We optimize the generator, classiﬁer and discriminator loss functions using Adam optimizer(Kingma & Ba (2014)).
Scaling
[Table]
Results show that augmenting real data with synthetic data improves classiﬁcation performance
E XPERIMENTS
Our work is motivated by the fact that fMRI data is rarely available.
To assess the suitability of generative training methods for data generation, in this section, we examine the performance of the two aforementioned models on three different datasets: collection 1952, 2138 and 503 1 .
To illustrate the beneﬁts of generated fMRI data for classiﬁer training, we focus on two variants of the model architecture for generating images at 13 × 15 × 11 and 26 × 31 × 23 spatial resolutions for collection 1952, and 13 × 15 × 11 and 26 × 31 × 22 resolutions for the other two collections.
In Section 3, we present the architectures of our models for generating low resolution data, which only have 3 convolutional layers.
For high resolution data we add one more convolutional and deconvolutional layer in the discriminator and generator respectively.
At the same time, for high resolution data the classiﬁer also has 4 convolutional layers in the ACD-GAN.
We ﬁrst show qualitative results of generated 3D volumes.
Then we provide quantitative results for 3D volume classiﬁcation.
To carefully assess the classiﬁer performance we use cross validation in all reported results.
See the supplementary material for details.
Dataset 1: Collection 1952 Collection 1952 has 6573 brain images, 45 classes with a total number of 19 sub-classes.
The labels include task description that subjects were going through during the imaging process, such as visual, language and calculate.
A combination of 2 to 4 sub-classes are subsumed for a class.
There are a total of 45 classes because not every potential combination is reasonable.
In our experiments, we work on a subset of 36 classes because 9 of the 45 classes have less than 30 examples.
If a class has more than 100 images, we split images in that class 7:1: into training, validation and test datasets.
Below 100 images but more than 30 images we use the proportion of 3:1:2 for splitting images.
We ensure that the test data always consists of real images only. We ensure that the test data always consists of real images only.
We train the ICW-GAN and
ACD-GAN for 2500 epochs with a learning rate of 1e-4 and 50% exponential decay for eachiterations and a batch size of 50. 2-D projections of several brain volumes generated by ICW-GAN
Scaling
Input
Classiﬁer
[Table]
Macro F
[Table]
are illustrated in Figure 3.
The projections in the top row of Figure 3 are brains of class ’visual words, language, visual’, and those at the bottom are from class ’places, visual, right hand’.
By examining the generated brain images, we ﬁnd diversity in both intra-class and between-class.
We refer the interested reader to the supplementary material for a comparison of generated brains and real brain recordings.
To assess the suitability of the generated data we conduct a classiﬁcation task.
Note that the test data is always composed of real images.
The classiﬁcation results are shown in Table
Inspired by Odena et al.
, we also compute the multi-scale structural similarity (MSSSIM) (Wang et al., 2003) to examine the intra-class diversity of the generated data.
MS-SSIM score is a well-designed similarity metric which utilizes image details at different resolutions.
Its values are within [0 , 1] .
Higher MS-SSIM values correspond to more similar images.
We measure the mean MS-SSIM score with 100 randomly chosen volumes of mixed data with a given class, and ones of real training data.
In 13 × 15 × 11 resolution, we ﬁnd that 22 classes with mixed data have a lower MS-SSIM score than only with real training data.
In other words, 61.1% classes with mixed data have sample variability that exceeds those only with real training data.
In 26 × 31 ×resolution, 69.4% classes with mixed data have a lower MS-SSIM score.
See Figure 4 for details.
Dataset 2: Collection 2138 For data in collection 2138, there are 1847 brain images, 61 classes and 50 labels.
Because of the small size of the dataset, we randomly choose 70% of the brain images as training data and leave 30% as test data.
In this case, we do not have development data to supervise the training process; thus, we train our models for 1000 epochs in several runs and record the best classiﬁcation results, which are summarized in Table 2.
In this collection, we downsample brains by a factor of 8 and 4 to volume sizes of 13 × 15 × 11 and 26 × 31 × 22 respectively.
Similar to the
Dataset
[Table]
earlier reported results we observe deep nets to outperform SVMs, but more importantly, generated data during training was again suitable to improve the classiﬁer performance on real data.
Dataset 3: Collection 503 5067 brain images are in collection 503.
Subjects in this dataset are required to respond to 30 images from the International Affective Picture Set (Chang et al., 2015).
These 30 images were used to train the Picture Induced Negative Emotion Signature also described in this paper.
We consider the 30 pictures classes of brain data.
In other words, there are overall 30 classes and each class represents a picture shown to subjects.
This collection needs extra preprocessing because brain volumes have two shapes: 79 × 95 × 68 and 91 × 109 × 91 .
We conform large volumes to the small ones.
Similarly, we do the experiment in the two levels of resolutions.
Classiﬁcation results are summarized in Table 3 and again follow the trend reported earlier.
Results of Using ACD-GAN We apply the ACD-GAN to brain data of low resolution which is efﬁcient and time-saving way to examine the model.
Since our ICW-GAN gains the best performance in the three databases if we use mixed training data, we compare these best results to results of ACD-GAN.
2-D projections of brain volumes generated by ACD-GAN are shown in Figure 5.
In
Table 4, we present classiﬁcation results of our models and show that ACD-GAN obtains improved accuracy scores compared to a ICW-GAN on most of the investigated metrics.
Comparison and Analysis To the best of our knowledge, we are the ﬁrst to apply generative models to artiﬁcially enlarge brain datasets and improve downstream classiﬁcation tasks.
As a result, we consider SVM classiﬁcation results as a baseline, no matter which sources of training data is used.
Though the distribution of each evaluation score for the three datasets is different, they hold two similar patterns.
First, according to all the results presented above, our ICW-GAN has overall better performance in these three datasets.
Further, the ICW-GAN hold the same status at both levels of resolution for brain images.
We only test the ACD-GAN in small resolution, results show that it can better improve the accuracy scores compared to ICW-GAN.
We also want to point out that our strategy, to mix real and generated data, although competitive, does not always yield the best results.
For example, using real data only to train the NN classiﬁer achieves the largest macro F1 score at high resolution in Table 3.
On the one hand, accuracy metrics are the most popular and outstanding score to estimate multi-class classiﬁcation tasks.
On another hand, to explain why our models cannot always achieve the highest score in other scores, we think there exists a reasonable range of variation for those scores because of a few unavoidable factors such as parameter initialization.
C ONCLUSIONS
Generative models provide a useful tool for understanding the individual variability of brain images.
The results of this manuscript show – to our knowledge for the ﬁrst time, that 3-D conditional
GANs, in particular our proposed ICW-GAN and ACD-GAN, can generate high quality diverse and task dependent brain images.
We hope our results inspire additional research on generative models for brain imaging data.
Beyond qualitative evaluation, we evaluate quantitative performance by using the generated images as additional training data in a predictive model – mixing synthetic and real data to train classiﬁers.
The results show that our synthetic data augmentation can signiﬁcantly improve classiﬁcation accuracy under various circumstances – a result which may be of independent interest.
Future work will be focused additional qualitative evaluation of the generated images by neuroscience experts and exploration of various applications.
We also plan to more throughly investigate the trained models to further explore what it may contribute to the science of individual variability in neuroimaging.
Finally, we plan to expand our models to combine data across multiple studies – each of which use different labels, by exploring techniques for merging labels based on the underlying cognitive processes (Poldrack, 2006).
S UPPLEMENTARY M ATERIAL 6. C ROSS -V ALIDATION
In Figure 6, the blue bar represents training data and the green bar is test data.
In our case, we construct a 3-round cross-validation task to measure performance of our models.
First, we partition test data to three folds and each fold maintains similar proportion of data for each class as original test data.
For each round of cross-validation, we leverage one of the three parts as test data and left two folds together with training data to train a model.
Each evaluation metric below is the average of the three rounds.
6. G ENERATED I MAGE A NALYSIS
We used NeuroSynth 2 , a brain decoding tool to qualitatively analyze generated volumes.
Its principle is to compare activity regions of a given brain volume to a data pool and then to obtain an analysis report about correlation to speciﬁc activity patterns.
Figure 7 shows a generated brain image and its analysis report with top 10 ranking list of correlation.
The supervised class of this generated brain is ”Non-human sound, auditory”, and several correlations with top rank are related to auditory and sound.
Analysis
[Figure]
[Figure]
We also examine our models in a more tough multi-class task.
As described before, one class in
Collection 1952 consists of several labels and the collection has totally 19 sub-labels.
Insetead of encoding the classes to one-hot vectors with 45 dimansions , we encode them in 19 dimensions, each of which represents a sub-label.
The value in a dimension is 1 only if the class possesses that label, otherwise, 0, i.e., one class has sub-label 0 and 1 being encoded as a 19 dimensional vector with the value of the ﬁst two dimensions 1 and 0 in other dimensions(formally present it as ( [1 , 1 , 0 , ..., 0]) ).
This task is highly demanding since that there are larger probability space, speciﬁcally, 2 19 possibilities, for a classiﬁer to learn and discriminate.
We use SVM to train real and mixed training data, both of which are in the resolution of 13 × 15 × 11 , and utilize 3 fold crossvalidation as illustrated in Section 6.1 to this multi-class task.
Several evaluation scores are shown [Table] in Table 2.
It is shown that even in this highly demanding task, accuracy score with mixed data(the second row) used is still visually higher than the baseline(the ﬁrst row).
6. R EAL BRAIN IMAGES OF C OLLECTION 1952 WITH LABELS ON TOP
We present several real images of Collection 1952 with their labels above the 2D projections of brain images in Figure 8.
6. M ORE BRAIN GENERATIONS
More synthetic brain images using ICW-GAN are shown in this section.
Figure 9 ∼ 11 are generated examples from Collection 1952, 2138 and 503 respectively.
For Collection 1952, classes are on the top of images.
As for Collection 2138, the corresponding categories of the classes are listed as follow:
• 14: visual form recognition, feature comparison, response selection, response execution, relational comparison, visual pattern recognition • 22: response execution, working memory, body maintenance, visual body recognition • 4: response selection, response execution, punishment processing • 43: motion detection • 2: response selection, response execution, animacy perception, animacy decision, motion detection • 3: response selection, response execution, motion detection