From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper19/Authors
comment: Hello,
Thank you very much for your constructive comments. I tried to provide answers for your observations:

1. Motivation:
-    Our motivation was indeed, to design a model capable of predicting the full set of voxels, rather than the subset of voxels as done with previous methods. To achieve this, we have tried various combinations layers, parameters (architecture), and "stimulus" which gave us better results and hence can be considered as our method.

2. Incomplete experiments:
-    We have performed an experiment with images as stimulus and observed lower performance (F1-score) when compared to embedding or multi-modal inputs. The lower performance can be attributed to the fact that the variation between images is very large when compared to the variation between word embedding.  We will accommodate image details in the final version of the paper.
-    We were not able to compare our results to any other existing methods since,
     i) The literature for fMRI encoding models is very sparse.
     ii) Earlier models considered only a subset of voxels and posed the problem as regression analysis, as described in 
Section 1.  Hence we were unable to use their references directly and had to mention them only in the introduction section.  This is the first time where 3D volume was converted to 2D slices and encoder-decoder architecture was applied to predict all the slices. This gave us two distinct advantages. a) we had more data in had to train a deep learning model and b) clear spatial and temporal markers across 2D slides which helped us to choose better architecture to learn and predict activations.

3. Figure 1 slice predictions:
-    We will replace the current image with one with better resolution and highlight the common activations using colored rectangles. We were not able to use the difference of images since we wanted to highlight activation predictions along with its location in brain slice, which is one the strengths of our paper.

4. Minor comments:
-     Thank you. The mentioned observations will be rectified in the final version of our paper.
title: Answers:

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper19/Authors
comment: Hello, 
Thank you very much for your constructive comments. I tried to answer your observation for question 4:

-      In our scenario, the model needs to capture the gradual variations in the activation, that occur across 70 fMRI slices.  The gradual variation can even be observed in Figure 1, which are 10 slices apart. Hence, two layers of LSTMs (256, 128) were used as latent layers since LSTM networks are capable of modeling sequential/temporal aspects of data better than dense or CNN layers. We even tried ConvLSTM and bidirectional LSTM instead of CNN and LSTM layers. Both the models gave similar results when compared to the architecture described in our paper, hence we report only CNN-LSTM based architecture results in the experiments section.

-     A single model per subject predicts 70 slices using just the multi-modal information (word, image). This was possible only by using LSTMs which is trained to "remember" 70 steps (batch size) before switching to the new concept, as shown in figure 2 (architecture). This proves that the temporal component is a necessity and was extensively used for training our model.

-     Pre-processing was indeed applied to the fMRI slices.  We normalized the pixel values using a min-max scaler to restrict the values between 0 and 1. But this required, similar to any deep learning architecture for better performance. No other form of pre-processing was applied to the fMRI slices.

title: Answers:

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper19/AnonReviewer1
evaluation: This paper proposes a method/computational model to visualize neural activity of the brain in response to external stimuli (e.g. images or text). Such a computational model can greatly benefit neuroscientists, which will enable to run experiments faster than relying on actual patients. The paper proposes a convolution neural network and long short-term memory (LSTM) based model to predict brain activity for a given stimuli.

The method uses a sequence of sliced images of the brain using all 3D voxels, which then uses a CNN based encoder/decoder architecture, to predict the current neural activity of the brain, given the stimuli and the history of brain activity. To store the temporal memory, an LSTM is used between the encoder and the decoder.

Overall, though this paper highlights the advantages of using multi-modal data in modelling brain activity, I feel the experiments need to be more self-contained. Additionally, the motivation of the paper is not clear. Please read the comments below for details on these remarks.

Strengths
--------------
+ The method shows promise, as the visualizations (Figure 1) and experiments show that the proposed method performs better when multimodal response is used compared to using GloVe word embeddings.

Major comments
--------------
I find the motivation of this paper dispersed. Authors need to have more clarity around whether they are,
1. Highlighting the advantage of using the full set of voxels, than the subset of voxels as done with previous methods (as said so in section 2) or,
2. Highlighting the advantage of using multi-modal data than unimodal data (as discussed in section 3).

The experiments are incomplete, 
1. For example I would have expected a comparison between multi-modal, text-only, and image-only. However, comparison between the multi-modal and text-only data exists.
2. Additionally, the method has not been compared to any other existing method.

In Figure 1, it is difficult to see how good the predictions are compared to the ground-truth. An image computed by difference between the two images would have been more helpful.

Minor comments
--------------
Sentence in lines 99-101 needs to be better formulated

Line 124-127: If the results related to perturbations are not reported, I would avoid mentioning about them in the text, as it is not adding any contribution to the paper.

Section 2 is not well composed. It needs to be broken down to two sections, related work, and our approach.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper19/AnonReviewer1
relevance: 4: high
impact: 3: borderline
novelty: 2: low

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper19/AnonReviewer2
evaluation:     1. What did you learn from reading this paper that you did not know before?
	The authors take a encoder-decoder CNN LSTM model and apply it to a novel application of fMRI slice sequences that learns a mapping of stimulus presented to the patient (words and images) to brain activation. Despite using a dataset of 16*180 (=2880) fMRI volumes, experiments show that the model was able to predict the intensities of voxels (the dimensionality of which is 128*88*85 (=957440)) with a high precision, recall and F1 score.
    2. What was the most exciting part of the work?
	That a 3D fMRI brain activation map (128*88*85) was sliced as 2D data along the axial plane (85 slices of 128*88 images)  and each slice was progressively fed into a CNN LSTM decoder/encoder network, could be used to indirectly model volumetric data. The "peturbation experiments" seems to be a promising result as well.
    3. What in your opinion makes this work stand apart from other papers in this area?
	Most previous work relies on the assumption that the brain activation is linearly related to the stimulus or some parametric transformation of it. This work uses a deep learning model instead.
    4. Does the paper fit into the scope of the workshop?
	Raw fMRI data has a temporal component, however the data used in experiments only uses volumetric data. The data used in experiments seems to have undergone significant preprocessing where the temporaral component of the data has been summarised into a brain activation map (spatial data, probably through a first level  GLM analysis). However it is unclear as the citation provided for this data does not seem to include any information how this preprocessing/analysis was carried out. Overall, the model proposed disregards the temporal component of the original data and hence may not suitable for a spatiotemporal workshop.
    5. Are you aware of relevant literature that you may want to make the authors aware of?
None

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper19/AnonReviewer2
relevance: 2: low
impact: 3: borderline
novelty: 4: high