H ALLUCINATING BRAINS WITH ARTIFICIAL BRAINS
Anonymous authors
Paper under double-blind review
A BSTRACT
Human brain function as measured by functional magnetic resonance imaging ( fMRI)
, exhibits a rich diversity
In response, understanding the individual variability of brain function and its association with behavior has become one of the major concerns in modern cognitive neuroscience.
Our work is motivated by the view that generative models provide a useful tool for understanding this variability.
To this end this manuscript presents two novel generative models trained on real neuroimaging data which synthesize task-dependent functional brain images.
Brain images are high dimensional tensors which exhibit structured spatial correlations.
Thus, both models are 3D conditional Generative Adversarial networks (GANs) which apply Convolutional Neural Networks (CNNs) to learn an abstraction of brain image representations.
Our results show that the generated brain images are diverse, yet task dependent.
In addition to qualitative evaluation, we utilize the generated synthetic brain volumes as additional training data to improve downstream fMRI classiﬁers (also known as decoding, or brain reading).
Our approach achieves signiﬁcant improvements for a variety of datasets, classiﬁcation tasks and evaluation scores.
Our classiﬁcation results provide a quantitative evaluation of the quality of the generated images, and also serve as an additional contribution of this manuscript
I NTRODUCTION
Functional Magnetic Resonance Imaging (fMRI) is a common tool used by cognitive neuroscientists to investigate the properties of brain function in response to stimuli.
Classic analysis approaches (Poldrack et al., 2011) focused on analyzing group-averaged brain function images.
However, it was discovered that brain activation patterns vary signiﬁcantly between individuals.
Thus, modern analysis now prioritizes understanding the inter-subject variability of brain function (Dubois & Adolphs, 2016; Geerligs et al., 2017).
Our work is motivated by the view that generative models provide a useful tool for understanding this variability – as they enable the synthesis of a variety of plausible brain images representing different hypothesized individuals, and high-quality generative models can be analyzed to posit potential mechanisms that explain this variability (Horn et al., 2008).
The results presented in this paper provide – to our knowledge for the ﬁrst time, positive results suggesting that it is indeed possible to generate high quality diverse and task dependent brain images.
While we can qualitatively evaluate generative brain images, quantitative evaluation allows us to objectively compare between various results.
To this end, we utilize the generated synthetic brain volumes as additional training data to improve downstream fMRI classiﬁers.
The use of classiﬁers to predict behavior associated with brain images is also known as decoding or brain reading (Varoquaux & Thirion, 2014; Pereira et al., 2009).
Classiﬁers such as support vector machines and deep networks have been applied for decoding brain images.
For example, Cox & Savoy (2003) attempted to classify which of 10 categories of object a subject was looking at (including similar categories, such as horses and cows) based on limited number of brain images.
Besides visual tasks, Mitchell et al.
(2008) distinguished active regions of brains when subjects listened to linguistic words, where the stimuli included ﬁve items from each of 12 semantic categories (animals, body parts etc.).
Beyond providing a model for individual variability, high quality brain image synthesis addresses pressing data issues in cognitive neuroscience.
Progress in the computational neurosciences is stiﬂed by the difﬁculty of obtaining brain data either because of a limited culture of data sharing, or due to medical privacy regulations (Poldrack & Gorgolewski, 2014).
For the computational neuroscientist, generated images deliver unlimited quantities of high quality brain imaging data that can be used to develop state of the art tools before application to real subjects and/or patients (Varoquaux & Thirion, 2014).
This approach of using modern generative models to synthesize data, which in turn accelerates scientiﬁc study, has already proven useful in many scientiﬁc ﬁelds such as particle physics and astronomy (Castelvecchi et al., 2017).
Our work represent a ﬁrst application for this approach to neuroscience.
One of the promising generative models are Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), capturing complex distributions using a non-cooperative two-player game formulation: a generator produces synthetic data by transforming samples drawn from a simple distribution; a discriminator focuses on distinguishing synthetic and real data.
Despite (or possibly due to) the compelling formulation, GAN training is known to be unstable.
To address this difﬁculty various variants have been proposed.
Wasserstein GANs (WGANs)
(Arjovsky et al., 2017) formulate the objective using the Wasserstein distance rather than the classical Jenson-Shannon divergence.
Improved training of WGANs (Gulrajani et al., 2017) applies an additional gradient penalty, which avoids the critical weight clipping in WGANs which might lead to pathological behavior.
Dualing
GANs restrict the discriminator and formulate the dual objective (Li et al., 2017).
Beyond improving the stability, conditional GANs (Mirza & Osindero, 2014) make it possible to control the data generation process by conditioning the model on additional information.
Auxiliary Classiﬁer GANs (AC-GANs) (Odena et al., 2016) unify a GAN and classiﬁer to a single architecture, employing labels to generate ImageNet samples.
3D GANs (Wu et al., 2016) reconstruct 3D objects and Edward & David (2017) propose to use improved WGAN to enhance the stability of 3D GANs.
We make the following contributions in this paper:
We develop Improved Conditional Wasserstein GANs (ICW-GAN) and Auxiliary Classiﬁer and Discriminator GANs (ACD-GAN), two types of 3D conditional GANs to synthesize fMRI brain data, both of which we ﬁnd to be stable to train.
We assess the qualitative quality and diversity of generated brain volumes.
Our results suggest that the proposed models are able to generate high quality task-dependent and diverse 3D brain images.
We evaluate our models on three datasets using a series of image classiﬁcation tasks with support vector machines and deep network classiﬁers at two levels of brain image resolution.
Results show that augmenting training data using synthetic data generated by our models can greatly improve test classiﬁcation accuracy of brain volumes.
R ELATED W ORK 2. N EURAL N ETS FOR F MRI B RAIN D ATA
We are unaware of any published papers using neural networks to generate brain imaging data.
However, neural networks have been used by several authors for classifying brain imaging data.
Firat et al.
(2014) and Koyamada et al.
(2015), used 2D deep nets to extract features of fMRI brain images to classify brain states.
Nathawani et al.
(2016) applied both 2D and 3D neural networks to classify fMRI brain data.
Svanera et al.
(2017) decoded fMRI data of video stimuli and classify data into visual categories.
Similarly, Nathawani et al.
(2016) extracted features from 4-D fMRI data and used deep learning methods for discrimination of cognitive processes.
2. G ENERATIVE A DVERSARIAL N ETS
To learn a distribution over data x , a GAN (Goodfellow et al., 2014) formulates a 2-player noncooperative game between two deep nets.
The generator G uses a random noise vector z sampled from a prior distribution P z ( z ) as input and produces an image G ( z ) .
The generator is trained to fool the discriminator, which receives either synthetic data or real data and differentiates between them.
Formally, G and D play the following two-player minimax game with value function V ( G, D ) :
[Equation]
For the conditional GAN (Mirza & Osindero, 2014), both G and D are conditioned on some extra information y , for instance, class labels or other features.
The conditioning can be presented by feeding y into both the generator and the discriminator.
This work is successful in one-to-many mappings such as image labeling with many tags.
The Wasserstein GAN (WGAN) (Arjovsky et al., 2017) uses Wasserstein-1 distance W ( q, p ) as the objective function.
It computes the minimum cost of transporting mass to transform the distribution q into the distribution p .
Under this circumstance, W ( q, p ) is continuous everywhere and its gradient with respect to its input was found to be more stable than its classical GAN counterpart.
Improved
Wasserstein GAN (IWGAN) (Gulrajani et al., 2017) argues that weight clipping of the critic, i.e., the discriminator, in WGANs inevitably causes the gradient to either vanish or to explode.
To address this issue Gulrajani et al.
(2017) propose an alternative penalty term in the critic loss based on the gradient norm.
In an auxiliary classiﬁer GAN (AC-GAN) (Odena et al., 2016), every generated sample has a corresponding label which can be used as a condition in the generator and the classiﬁer .
The discriminator is modiﬁed to contain an auxiliary decoder network to reconstruct the training data class.
Further, 3D-GANs (Wu et al., 2016) extend GANs to 3D object generation.
Different from classical
GANs, 3D-GANs apply the three dimensional convolution in both G and the D. By learning deep object representations, 3D GANs can generate visually appealing yet variable 3D object volumes.
A PPROACH
In the following we introduce two models for fMRI data generation: ICW-GAN and ACD-GAN.
They differ in model structure and in the way label information is taken into account.
We ﬁrst show the model structure of 3D ICW-GAN and its downstream classiﬁers in Section 3.1, and present
ACD-GAN in Section 3.2.
3. 3D I MPROVED C ONDITIONAL W ASSERSTEIN GAN (ICW-GAN)
Similar to classical generative adversarial networks (GANs), ICW-GANs are formulated as a noncooperative two-player game between two adversaries: (1) a generator ˆ x =
G θ ( z ) , which generates artiﬁcial samples ˆ x from randomly drawn latent encodings z via a transformation using a deep net parameterized by θ ; and (2) a discriminator D w ( x ) represented via the logit obtained from a deep net parameterized by w .
Following Arjovsky et al.
, we build our models based on minimizing the Wasserstein distance via
[Equation]
where { D w } w ∈ W denotes a set of functions that are K-Lipschitz for some K. IW-GANs (Gulrajani et al., 2017) provide a more stable method for enforcing the Lipschitz constraint by employing a gradient penalty rather than weight clipping, when optimizing:
[Equation]
Hereby ˆ x is a convex combination of real data and artiﬁcal samples, i.e., ˆ x ← (cid:15) x
+ (1 − (cid:15) )
G ( z ) with (cid:15) drawn from a uniform distribution ( (cid:15) ∼ U [0 , 1] ).
In our models, we extend IW-GAN in two directions.
First, because fMRI data is three dimensional, 3D convolution and deconvolution are used in order to maintain the spatial structure and voxel information of the fMRI data.
Second, we condition both the discriminator D w and the generator
G θ on available labeled data.
As shown in Figure 1, our generator consists of three fully convolutional layers.
In convolutional layers, we use kernels with a size 4 × 4 × 4 and strides 2, with batch normalization and Leaky ReLU layers added between and a tanh layer at the end.
The discriminator architecture is a mirrored generator, but no activation function is used at its end.
This ICW-GAN model follows models proposed in (Gulrajani et al., 2017; Mirza & Osindero, 2014;
Wu et al., 2016), however, we found the following modiﬁcations to stabilize training and supervise the generation process for classiﬁcation data: compared to existing GANs, our ICW-GAN applies a more stable upper bound for Wasserstein distance as an objective function, leverages conditioning labels to supervise the generation process, and utilizes 3D convolution methods for 3D object generation.
To include label information, we concatenate labels to the input and to the hidden layers.
At the input of the generator, one-hot labels are combined with the brain vector.
Then, for each of the intermediate layers, we use a fully connected layer followed by a tanh activation to transform the one-hot vector to a volume of appropriate size, i.e., 15 × 2 × 2 for the ﬁrst hidden layer, and 15 × 3 × 3 for the next.
We empirically found dimensions other than 15 , e.g., 3 to work well too.
We concatenate the label volume to intermediate volumes on feature dimension and pass the joint one to the next deconvolution layer.
We follow the same procedure in the architecture of the discriminator.
Referring to the volume labels via y , the objective function L of our ICW-GAN model is as follows:
[Equation]
z ∼ P z ( z ) [(1 − D ( G ( z | y )))] − E x ∼ P data ( x ) [ D ( x | y )]+ λ E ˆ x ∼ P ˆ x [( (cid:107) ∇ ˆ x D ( ˆ x | y ) (cid:107) 2 − 1) 2 ] , (4) where ˆ x ← (cid:15) x
+ (1 − (cid:15) )
G ( z ) , (cid:15) ∼ U [0 , 1] .
λ is a gradient penalty coefﬁcient.
We optimize both the discriminator and generator loss using Adam optimizer (Kingma & Ba, 2014).
D OWNSTREAM C LASSIFIERS FOR ICW-GAN
Our work is partially motivated by the difﬁculty of acquiring fMRI data, so we consider the task of enlarging the dataset for downstream tasks such as classiﬁcation, using synthetically generated samples.
To assess the suitability of the developed generation procedure, we employ a variety of downstream classiﬁcation processes.
More speciﬁcally, we train a classiﬁer with real (‘Real’) or real plus generated (‘Real+G’) data.
Common classiﬁers in neuroscience are Support Vector Machines (SVMs) and 3D deep networks.
We think usage of both provides valuable insights as their number of trainable parameters differ signiﬁcantly, i.e., 14 parameters for SVMs and several hundred thousand for our deep networks.
To measure the results obtained from different training datasets we use accuracy, macro F1, precision and recall metrics.
SVM : We use a simple linear SVM to
classify test data and don’t
extract any
intermediate features. Instead, we use raw brain data, vectorized to a 1-dimensional vector for classiﬁcation to avoid any form of collusion Deep
Net :
The deep net structure is similar to the discriminator, having a 3 dimensional structure and identical number of convolution layers with a Leaky ReLU activation. Obviously the classi- Downsampling
[Table]
Recall
[Table]
0 × [Table] means training synthetic data which is produced by Two are utilized: SVM and neural networks (NN) that augmenting real data with synthetic improves performance ﬁer doesn’t concatenate intermediate and input data with any label information.
We provide the respective information for each dataset in the experimental section.
3. 3D A UXILIARY C LASSIFIER AND D ISCRIMINATOR GAN (ACD-GAN)
We also develop a conditional GAN architecture which jointly trains the generator, the discriminator and the classiﬁer.
This form of training is inspired by the AC-GAN (Odena et al., 2016) formulation.
Our generator architecture is illustrated in Figure 1 and identical to the one employed for the ICW-GAN.
The combination of classiﬁer and discriminator is provided in Figure 2.
As before, the discriminator assesses the logit of the probability that a provided sample is real or artiﬁcially generated.
In addition, the classiﬁer provides a probability distribution over the class labels.
In this model, training passes labels through the generator and the discriminator in order to condition the adversarial process.
However, no labels are employed in the classiﬁcation stream.
The classiﬁer loss, L C , is the log-likelihood of the correct class.
Compared to L in Eq.
(4), the discriminator loss, L D includes the additional classiﬁer component
[Equation]
The generator loss function, L G , consists of the likelihood of fake images and one part of L C , i.e.,
[Equation]
Similarly, the discriminator loss function is given by L D =
L + L C .
We optimize the generator, classiﬁer and discriminator loss functions using Adam optimizer (Kingma & Ba, 2014).
[Table]
E XPERIMENTS
We examine the performance of the two aforementioned models on three different datasets from Neurovault (Gorgolewski et al., 2015): collections 1952, 2138 and 503 1 . To evaluate the beneﬁts of generated fMRI data for classiﬁer training, we focus on two variants of the model architecture for generating images.
The original images are downsampled using the nilearn python package 2 to 13 × 15 × 11 (lower resolution) and 26 × 31 × 23 (higher resolution) spatial resolutions for collection 1952, and 13 × 15 × 11 (lower resolution) and 26 × 31 × 22 (higher resolution) resolutions for the other two collections.
In Section 3, we present the architectures of our models for generating low resolution data, which only have 3 convolutional layers.
For high resolution data we add one more convolutional and deconvolutional layer in the discriminator and generator respectively.
For high resolution data, the classiﬁer also uses 4 convolutional layers in the ACD-GAN.
We ﬁrst show qualitative results for generated 3D volumes, then we provide quantitative results for 3D volume classiﬁcation.
We use cross validation in all reported results to carefully assess the classiﬁer performance.
See the supplementary material for details.
Dataset 1: Collection 1952 Collection 1952 has 6573 brain images, 45 classes with a total number of 19 sub-classes.
The labels include task description that subjects were going through during the imaging process, such as visual, language and calculate.
A combination of 2 to 4 sub-classes are subsumed for a class.
There are a total of 45 classes because not every potential combination is observed.
In our experiments, we work on a subset of 36 classes because 9 of the 45 classes have less than 30 examples.
If a class has more than 100 images, we split images in that class 7:1:2 into training, validation and test datasets.
Below 100 images but more than 30 images we use a 3:1: Downsampling Input [Table]
Precision
Recall [Table] data split. Note that we ensure that the test data always consists of real images only.
We train the ICW-GAN and ACD-GAN for 2500 epochs with a learning rate of 1e-4 and 50% exponential decay for each 3000 iterations and a batch size of 50.
2-D projections of several brain volumes generated by ICW-GAN are illustrated in Figure 3.
The projections in the top row of Figure 3 are brains of class ‘visual words examining refer interested reader to
Inspired by Odena et al.
, we compute the multi-scale structural similarity (MS-SSIM) (Wang et al., 2003) to examine the intra-class diversity of the data generated by ICW-GAN.
The MS-SSIM score is a similarity metric based on image similarity at different resolutions.
Its values are within [0 , 1] .
Higher MS-SSIM values correspond to more similar images.
We measure the mean MS-SSIM score with 100 randomly chosen volumes of mixed data with a given class, and ones of real training data.
In 13 × 15 × 11 resolution, we ﬁnd that 22 classes with mixed data have a lower MS-SSIM score than only with real training data.
In other words, 61.1% classes with mixed data have sample variability that exceeds those only with real training data.
In 26 × 31 × 23 resolution, 69.4% classes with mixed data have a lower MS-SSIM score.
See Figure 4 for details.
To further assess the quality of the generated data we evaluate classiﬁcation performance.
Note that the test data is always composed of real images.
The classiﬁcation results are shown in Table 1.
Downsampling refers to the downscaling factor as already discussed.
The second column indicates the type of training data we use for the classiﬁer: only using real data (‘Real’), or using the mixed data of real and generated volumes (‘Real+G’). The third column denotes the classiﬁer type, i.e., an SVM or a deep net (‘NN’). We use the validation dataset to choose the best training models and use these models to classify the test data.
We observe the deep net classiﬁer to generally outperform
SVMs and we observe generated data to be beneﬁcial for classiﬁer training.
Dataset 2: Collection 2138 For data in collection 2138, there are 1847 brain images, 61 classes and 50 labels.
Because of the small size of the dataset, we randomly choose 70% of the brain images Dataset [Table] as training data and leave 30% as test data.
In this case, we do not have development data to supervise the training process; thus, we train our models for 1000 epochs in several runs and record the best classiﬁcation results, which are summarized in Table 2.
In this collection, we downsample brains by a factor of 8 and 4 to volume sizes of 13 × 15 × 11 and 26 × 31 × 22 respectively.
Similar to the earlier reported results we observe deep nets to outperform SVMs, but more importantly, generated data during training was again suitable to improve the classiﬁer performance on real data.
Dataset 3: Collection 503 5067 brain images are in collection 503.
Subjects in this dataset are required to respond to 30 images from the International Affective Picture Set (Chang et al., 2015).
These 30 images were used to train the Picture Induced Negative Emotion Signature also described in this paper.
We consider the 30 pictures classes of brain data.
In other words, there are overall 30 classes and each class represents a picture shown to subjects.
This collection needs extra preprocessing because brain volumes are available with two shapes: 79 × 95 × 68 and 91 × 109 × 91 .
We conform large volumes to the small ones.
Similarly, all experiments are at the two levels of resolution.
Classiﬁcation results are summarized in Table 3 and again follow the trend reported earlier.
Results of Using ACD-GAN Since our ICW-GAN achieves the best performance in the three databases when we use mixed training data, we compare these best results to results of ACD-GAN at the low resolution.
2-D projections of brain volumes generated by ACD-GAN are shown in Figure 5.
In Table 4, we present classiﬁcation results of our models and show that ACD-GAN obtains improved accuracy scores compared to a ICW-GAN for most of the investigated metrics.
As a baseline, we also implemented the recent generative model – the 3-D AC-GAN Odena et al.
According to all the results presented above, the proposed methods achieve much improved overall better performance on the three datasets.
This is true at both levels of resolution for brain images.
The results also outperform a 3-D AC-GAN applied to the same dataset.
We also observe that our strategy of mixing real and generated data does not always yield the best results.
For example, using real data only to train the NN classiﬁer achieves the largest macro F1 score at high resolution in Table 3.
However, the proposed approach consistently improves accuracy metrics – which is by far the most popular metric for evaluating multi-class classiﬁcation performance.
C ONCLUSIONS
Generative models provide a useful tool for understanding the individual variability of brain images.
The results of this manuscript show – to our knowledge for the ﬁrst time, that 3-D conditional
GANs, in particular our proposed ICW-GAN and ACD-GAN, can generate high quality diverse and task dependent brain images.
We hope our results inspire additional research on generative models for brain imaging data.
Beyond qualitative evaluation, we evaluate quantitative performance by using the generated images as additional training data in a predictive model – mixing synthetic and real data to train classiﬁers.
The results show that our synthetic data augmentation can signiﬁcantly improve classiﬁcation accuracy – a result which may be of independent interest.
Future work will focus on additional qualitative evaluation of the generated images by neuroscience experts and exploration of various applications.
We also plan to more throughly investigate the trained models to further explore what it may contribute to the science of individual variability in neuroimaging.
Finally, we plan to expand our models to combine data across multiple studies – each of which use different labels, by exploring techniques for merging labels based on the underlying cognitive processes (Poldrack, 2006).
S UPPLEMENTARY M ATERIAL 6. C ROSS -V ALIDATION S TRATEGY
First, we explain our cross-validation strategy.
In Figure 6, the blue bar represents training data and the green bar is test data.
For 3-fold cross-validation, ﬁrst we partition test data to three folds and each fold maintains a similar proportion of data for each class as original test data.
For each round of cross-validation, we leverage one of the three parts as test data and left two folds together with training data to train a model.
The same strategy is used with generated data to ensure that test performance is only based on the test data.
Each evaluation metric is the average of the three rounds.
6. V ARIANCE OF C ROSS -V ALIDATED P ERFORMANCE
Revision 1: We tested our model with various cross-validation settings and calculated the variances of the evaluation metrics (Table 5).
We conducted this experiment on low resolution data from collection 1952 and synthetic data generated by the ICW-GAN.
Except for the number of folds used when partitioning the test dataset, the training strategy for 5-fold and 10-fold cross-validation is similar to that of 3-fold cross-validation.
The small variances suggest that the reported accuracy differences, while small, are indeed signiﬁcant.
[Table]
10-fold cross validation.
We conducted this experiment with the training data of mixed ‘Real+Synth.’
data of collection 1952 in the low resolution (4.0 × ) setup.
6. T RAINING L OSS C URVE OF ICW-GAN IN V ALIDATION D ATASET IN C OLLECTION
Revision 2: We present the ICW-GAN training loss curve evaluated on the validation dataset of collection 1952 in Figure 7.
This illustrates the stability of the proposed improved Wasserstein variants.
[Table]
6. R ESULTS OF ONLY USING GENERATED DATA FOR TRAINING
Revision 3:
In order to further evaluate the quality of the generated data for collection 1952, we trained using only generated data and tested with real data.
We used the deep net classiﬁer for evaluation in this case while varying the number of input samples for each class.
Figure 8 shows that in general, the test accuracy improves as the amount of artiﬁcial training data increases.
The red and yellow lines show the accuracy obtained when only using real data.
These results suggest that the generated data can be as effective for classiﬁcation as real data with sufﬁcient numbers of samples.
Revision 4: We also used a Gaussian Mixture Model (GMM), i.e., trained separately for each class, as a simple baseline generative model.
The results are shown in Table 6, where 6 training strategies were listed and the deep net classiﬁer (same as the NN classiﬁer) was used for all the strategies.
I.e., we leveraged 20 synthetic images in each class (20 × number of classes in all) to train the NN classiﬁer.
Note that both generative models learned the probability distribution from collection 1952.
The evaluation scores of our models are signiﬁcantly higher than the ones obtained with a GMM, particularly when using only synthetic data to train the classiﬁer.
6. G ENERATED I MAGE Q UALITATIVE A NALYSIS
We used NeuroSynth 3 , a brain decoding tool to qualitatively analyze generated volumes.
Its principle is to compare activity regions of a given brain volume to a data pool and then to obtain an analysis report about correlation to speciﬁc activity patterns.
Figure 9 shows a generated brain image of collection 1952 by the ICW-GAN in low resolution and its analysis report with the top 10 ranking list of correlation.
The supervised class of this generated brain is “Non-human sound, auditory“, and several correlations with top rank are related to auditory and sound. [Figure]
heschi
[Figure]
sounds
[Figure]
fractional anisotropy [Figure] [Figure] pitch
[Figure]
anisotropy fa
[Figure] anisotropy [Figure] 6. A DDITIONAL M ULTICLASS C LASSIFICATION R ESULTS
We also examine our models using a multilabel classiﬁcation strategy in low resolution data of collection 1952.
As described before, one class in collection 1952 consists of several labels and the collection has a total of 19 sub-labels.
Instead of encoding the classes with a one-hot vector [Table] of 45 dimensions, we encode them in 19 dimensions, each of which represents a sub-label.
The value in a dimension is 1 only if the sample possesses that label, otherwise, 0, i.e., a sample comes with a 19 dimensional binary vector.
This representation is demanding since the probability space is signiﬁcantly larger, speciﬁcally, 2 19 possibilities, for a classiﬁer to learn and discriminate.
We use SVM to train real and mixed training data (synthetic data obtained from ICW-GAN), both of which are in the resolution of 13 × 15 × 11 , and utilize 3-fold cross-validation as discussed in Section 6.1.
Several evaluation scores are shown in Table 7.
We observe that even in this highly demanding task, accuracy scores with mixed data (the second row) outperform the baseline (
6. R EAL BRAIN IMAGES OF C OLLECTION 1952 WITH LABELS ON TOP
We present several real images of collection 1952 with their labels above the 2D projections of brain images in Figure 10.
6. A DDITIONAL S YNTHETIC B RAIN I MAGES
More synthetic brain images using ICW-GAN are shown in this section.
Figures 11-13 show generated ones of low resolution from collection 1952, 2138 and 503 respectively.
For collection 1952, classes are given above the images.
As for collection 2138, the corresponding categories of the classes are as follows:
• 14: visual form recognition, feature comparison, response selection, response execution, relational comparison, visual pattern recognition • 22: response execution, working memory, body maintenance, visual body recognition • 4: response selection, response execution, punishment processing • 43: motion detection • 2: response selection, response execution, animacy perception, animacy decision, motion detection • 3: response selection, response execution, motion detection