From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/Authors
title: Thanks for the discussion and correct supplement link
comment: Thank you for this discussion. 

I think I understand your point. Information of earlier sequence elements are processed more often and are thus abstracted to a higher level than later sequence elements.

Thank you for your comments and the notice of the broken link.

This link to the supplementary material should work: https://s3.eu-central-1.amazonaws.com/corupublic/NIPS18ST/NIPS18ST_suppliment.pdf

From: ~Frederik_Kratzert1
comment: Thank you again for your answer, I have no further questions.

Regarding point 3) You are of course right, the a one-directional LSTM introduces an bias, the question I had was, if this bias maybe is helpful in this task of predicting crops. My view is/was that different plants have different growth-cycles and this natural bias might be reflected and learned in an one-directional LSTM and could potentially be helpful. 

Unfortunately I don't know of much literature regarding this topic. There is "Adaptive Computation Time for Recurrent Neural Networks" by Alex Graves, which is related but already more complicated as what I wanted to say. 
But let me try to explain what I meant with different words: In the last time step you add new information into the system, that can possibly update the cell states. Most probably you might have many cell/hidden units, that are not only influenced by the current input but also (mainly) by the hidden output of the last time step. But because it is the last time step, you do not compute the hidden to hidden recurrent connections again and therefore do not update these cell states with the latest information. 
Therefore the input of the last time step only reaches the cell states through the input to hidden connections but not through the hidden to hidden connections. Earlier time steps are multiple times processed through the recurrent connection, thus have more importance. 
I hope this makes my remark more understandable.

P.S. The link to your supplementary material seems to be broken (at least I get the following error "Error: Unable to find URL to redirect to.")
title: Final comment

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/Authors
title: Answers to your follow-up questions
comment: Thank you for your follow-up questions. We apologize for only partly answering your previous remarks and the late response, as our initial response coincided with incoming reviewer evaluations.

Regarding 1) We agree that we should have included that information in the submitted manuscript. Based on your comments and the reviewer feedback we have become aware that we have not stated that clearly in the current version of the manuscript and we will refine the text in the final version.

Regarding 3) In the initial implementation of the network we straightaway implemented a bi-directional LSTM and did not perform additional experiments on one-directional encoding. Based on our literature review, we see bi-directional encoding as the most common strategy, especially since it is not associated with additional weights. I agree with your argumentation regarding the importance of the last samples. However, we believe that our argument is still valid since a one-directional encoding still introduces a sequential bias compared to the symmetric bi-directional encoding. We would be interested to know more and would appreciate if you could share literature or empirical evaluations on this issue. 

Regarding the activations. Yes, these activations are correct and generated in an automated process. The activations also show the forward pass only which explains the low activations at first elements of the sequence. 
As described in the paper, we find for most of the 256 feature maps per gate are difficult to visually interpret. Features are likely encoded in a combination of feature maps while we can only visualize one at a time.
Considering this we would like to emphasize our experimental result that the task of cloud masking seems to be concentrated on only a few feature dimensions that are visible in one feature map. 
We show more activations in the supplementary material of this work (https://tinyurl.com/NIPS18ST-supplement) and will be able to publish the extraction script with the source code with the publication of this work.

Regarding the specific activations.
- t_26 we find it reasonable that one the entire feature map returns zero. The RGB image seems covered entirely by clouds and other feature depth may not be set to zero.
- t_1 is likely all zero, as the hidden states are initialized as zero. This seems to for the next observations. Overall we see for the other activations in the supplementary material that the spatial detail in the feature maps increases over time which we interpreted as the aggregation of more high-level features.
- t_34: It is difficult to say why this cloud has not been masked. It is located on the rightmost edge of the image which may have contributed. Additionally, the cloud in t_34 seems not mapped in the input gate but causes the modulation gate j to switch sign. I find this a quite interesting occasion that seems to emphasize the difficulty to visually interpret these activations.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/Authors
comment: Dear Reviewer,

Thank you for your evaluation, your comments and perspective on our work. 

First of all, we clearly emphasize that none of the reported results have been evaluated on the training dataset.
Figure 3 shows accuracies on the validation dataset that were recorded in parallel in the training process. We apologize for any confusion in this context and clearly state the independence of the shown results from the trained parameters.
We chose this illustration since we wanted to demonstrate the consistency over the entire training process.
In this experiment, all networks had the same network topology and were trained on varying cloud cover-datasets.
Since only the dataset was changed between these runs, we report accuracies on the validation dataset, and we did not vary any network parameters, we argue that our experimental methodology is sound.

According to your suggestions we will also revise and clarify formulations of our methods section. However, Figure 1 shows a detailed depiction of the network architecture:
The used network consists of two components: (1) A one-layer ConvLSTM that encodes a variable-length input sequence to a fixed representation cT, illustrated on the left, and (2) a convolution that compresses the depth of cT to the number of classes at the right side.
The input images xt of size 24px by 24px by 13 bands are presented to the network in a bi-directional way: in sequence and in a reversed sequence.
Figure 1 only shows the in-sequence pass to for better visibility. The cyclic connections from ct to ct-1 and ht to ht-1 indicate iterative recurrent connections. To further clarify this encoding strategy, we will add an “unrolled” illustration diagram to Figure 1 in a revised version of the manuscript. 
The second convolutional part of the network is illustrated on the right side.  It compresses the depth of cT to the to the number of classes to produce softmax-probabilities for each of the 17 classes per pixel.

Overall, this configuration yields a straightforward network architecture with 3 topology-related hyper-parameters: (1) the depth of the hidden tensors d=256 that we determined by two-fold cross-validation on training and validation datasets. This depth is indicated in blue at all hidden tensors in Figure 1. To emphasize this, we will add annotations for height h, width w, and depth d to the tensors in this Figure.
The other two hyper-parameters are the sizes of the convolutional kernels  (2) in the ConvLSTM and (3) of the classification convolution.
We fixed these kernels to 3x3 pixels as it is a common default choice for CNNs.
We argue that the simplicity of the network design makes our results transferable to other applications and our the reported robustness regarding temporal noise relevant for other researchers.

Initially, we have experimented with more complex CNN+ConvLSTM configurations. The two tested CNN encoders used stacked conv+bn+relu layers that were residually and densely connected. In these first experiments, we found that the gradients initially skipped many of the CNN layers. It seemed that most of the feature extraction work was done within the ConvLSTM. So we decided to simplify the network by dropping the CNN encoder early in the design process.
Considering the large pixel size of 10m of this satellite imagery, it seems logical that large-scale spatial features play a minor role in the classification. The distinct temporal or spectral characteristics of vegetation seem to be better captured by the recurrent network part where the convolutions integrate additional neighborhood information.

This work presents additional evaluations on the robustness of ConvLSTM to cloud coverage. It extends prior work published of a remote sensing journal which focused on vegetation monitoring for an Earth observation-related audience. In the public version of this manuscript we will be able to reference the comparison with other crop classification approaches in detail.
These results showed that our straightforward ConvRNN architecture achieved state-of-the-art accuracies on a larger number of classes than competitors while using minimal preprocessing and no additional cloud filtering.
We would like to emphasize that after extensive literature review we are not aware of other approaches in remote sensing that can perform land cover classification without explicit cloud filtering on optical data. 
While we are presenting remote sensing-related results, we are confident that these results generalize to other use-cases where temporal noise inherent to the data acquisition thanks to the straightforward nature of our network topology.

Again, thank you for your comments on the perception of your work. 
We are looking forward to clarifying further questions and providing follow-up information.

Best regards,

The Authors
title: Addressing your concerns and providing additional information on method and experimental design.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/Authors
comment: Dear Reviewer,

Thank you for your precise summary of our work and your insightful comments and perspective.

This work presents additional evaluations on the robustness regarding cloud coverage. We believe our results would be of interest for a broader methodically oriented audience since noise in temporal data is a relevant topic for more research fields.
It extends prior work published of a journal article which focused on vegetation monitoring for an Earth observation-related audience. In that work, we were able to explain our experimental design and methodology in more detail. We apologize for confusion arising from unclear descriptions of the methodology and experimental design and will revise the text a the final public version.

We would like to clarify our methodology and address your concerns:

1) Imagery from the Sentinel 2 A and B satellites is gathered every three to ten days depending on the current orbit configuration. Image acquisitions are always around 10 am local time due to the sun-synchronous orbit. We agree that a temporal correlation with acquisition time is possible since we observe more cloud coverage in the winter months (begin and end of the sequence) than in the summer months (middle of sequence) in our region of interest. We anticipated this correlation and decided to randomly sample a subsequence of 30 images from all available 46 images for each sample at training and testing time. We took care that the sequential order of observations was maintained. This random drop-out of some input elements was designed to ensure that cloudy observations are presented to the network at different indices of the sequence. We designed this to counteract a temporal overfitting on cloudy observation at sequence indices.

2) We clearly state that all presented results are not based on the training dataset and apologize for any confusion arising regarding the train/valid/test split. Figure 3 shows accuracies on the validation dataset that have been recorded in parallel in the training process. We clearly state the independence of the shown results from the trained parameters. We chose this illustration since we wanted to demonstrate the consistency over the entire training process. In this experiment, all networks had the same network topology and were trained on varying cloud cover-datasets. Since only the dataset was changed between these runs, we show validation accuracy, and we did not vary any network parameters, we argue that our experimental methodology is sound.
We also ensured a spatial independence between training, validation and evaluation datasets by subdividing the area of interest in rectangular blocks of 3.84km by 3.84km with margins of 480m between the blocks. This process has been recommended in related work [7] that experimentally showed that randomly assigning individual samples to train/validation/test would introduce a spatial correlation.

3) We appreciate your questions regarding the training methodology and are happy to provide further detail.
The area of interest used in this work covers approximately 100km by 40km. With ground pixel sizes of 10m at this results in images of 10k by 4k pixel in size. We subdivided these images in tiles of 24px by 24px and 48px by 48px. The 24px tiles have been used for training and quantitative evaluation, while qualitative evaluation was inferred on the larger 48px tiles for better interpretability. This yielded approximately 70k 24px-tiles with 30k for training, 20k tiles for validation and 20k for evaluation.
The Sentinel 2 satellite measures 13 spectral bands at each pixel. So overall, the training dataset contains 10 Billion data points: 46 observations x 30k tiles x 24 pixel width x 24 pixel height x 13 spectral bands. These are used to learn approximately 2.5 Million parameters. LSTMx + LSTMh =  (13 inputs x 256 hidden depth x 4 gates x 3x3 conv kernel) + (256 hidden depth x 256 hidden depth x 4 gates x 3x3 conv kerne)l.
Summarizing, we optimize approximately 2.5 Million unknown parameters with 10 Billion data points.
We trained this network for 60 epochs on a Nvidia-P100 server with 16GB VRAM which took 52 hours. 
	
4) We understand this concern. We originally wanted to indicate different degrees of cloud coverage by varying shades of blue. We will use a visually more distinct palette in the future.

We highly appreciated your perspective on our work and your comments. It has provided us with a good indication of the perception of our work and identified important topics to be clarified. 
We are looking forward to answering further questions and providing follow-up information.

Best regards,

The Authors
title: To address your comments and concerns

From: ~Ransalu_Senanayake1
comment: Dear authors,

You should now be able to see the reviews. We encourage you to engage in the discussion by commenting on the reviews. Please attempt to clarify the issues highlighted by the reviewers.  

Regards,
Ransalu, Neal, Fabio, and Girish
title: Rebuttal

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/AnonReviewer1
relevance: 5: very high
impact: 4: high
novelty: 3: borderline

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/AnonReviewer1
evaluation: The paper presents a convolution LSTM network to classify vegetation in (partially) cloud obstructed satellite imagery. The presented method eliminates the requirement for pre-processing these satellite images, as is the case for conventional methods. The paper presents a sound LSTM based methodology  to tackle the problem, but the results section needs more work.
Specific questions are given below.
1) Whats the time difference between two snapshots in a sequence of input images? Since the network is trained on satellite imagery, I’d imagine the time difference to be in the order of hours if not days. Would there be any temporal correlation in the cloud cover over such large time periods? This information is critical for the correct evaluation of the impact of the paper.

2) The paper presents only training performance, not any testing performance. Maybe the network is overfitting to the training data. Without testing performance information, it is very difficult to judge the effectiveness of the method. This also ties back to my concern in 1) above. There might not be any temporal correlation in a given sequence of images, due to the large time gap, and the network might be overfitting the training data.

3) It seems only 46 image sequences were used for training. It seems to be quite small considering the number of parameters to be learned. Maybe I am not understanding your training methodology properly. Either way, I would like to see some elaboration regarding the training data.

4) The colors in Figure 3 are almost indistinguishable, which makes it quite difficult to understand section 4.2. Please use different colors instead of different shades of blue.

From: ~Frederik_Kratzert1
comment: Thanks for your detailed answer but some points remain unclear to me:

regarding 1) I'm glad to hear that you did data splitting by it is unclear to me why this is never stated in your manuscript, as it is an important information. Especially if you now report the accuracy on the test set, you have to give details about the data splitting you performed (which sounds well thought and reasonable).

regarding 3) Maybe my question was unclear but what I wanted to know is if you, during your research, also tried to train a ConvLSTM that is _not_ bi-directional and compared the performance against your bi-directional approach. a) it is not totally clear that ConvLSTM (or LSTMs in general) have a bias towards the last samples in the sequence. Infact the last sample might even provide the least information, because the input of the last sample was never processed through the recurrent connections of the LSTM. However, this is another topic. Here I would be more interested in empirical results thus the question if you tried a non-bidirectional approach as well.

Beside some spelling corrections I also asked some questions under minor comments that you didn't answer:

- Is the image for j and i correct at t_26? Is there not a single pixel where the input gate is open? Same for t_1
- Why do you think, that the clouds at t_34 are not detected by the input gate?
title: Further questions

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/Authors
comment: Dear Frederik Kratzert,

Thank you for reading our submission and providing a new perspective on the perception of our work. 
We highly value your feedback and would like to address your comments and concerns. 

First, we firmly state the methodical soundness of our work and would like to address concerns regarding data partitioning: 
We have separated our Area of Interest (AOI) into partitions for the training/validation and evaluation to ensure independence between the optimization of network parameters, hyper-parameter tuning, and final evaluation.
Further, we took special care to avoid introducing dependence between these partitions by spatial proximity.
To ensure this, we subdivided the AOI into rectangular blocks of 3.84 km × 3.84 km (multiples of the chosen tile sizes 240 m and 480 m) with margins between blocks of 480m.
These blocks were randomly assigned to the partitions in a 4:1:1 train/valid/eval ratio.
All tiles (240m and 480m) that were fully within the respective blocks have been assigned to their respective partition. 
We acknowledge that this confusion was likely introduced by Figure 3. The shown curves show accuracies on the validation dataset, not on the training dataset. We evaluated the accuracy on this validation dataset in parallel to the training of the network.
We aimed to demonstrate with these curves that the overall accuracy of all runs was similar consistent during the entire training process, which proves the robustness of the method.

This work focused on presenting results on the robustness of ConvLSTMs with regard to temporal noise.
In particular, we show that the extent of clouds has been identified without explicitly providing cloud labels. This is novel to the field of remote sensing and the robustness of the ConvLSTM models to noise can serve as an example for other research domains.
We are confident that these results generalize well to other use-cases where temporal noise inherent to the data acquisition and that the particular application (i.e. crop classification) is not substantial to the findings presented in this work.
The task of crop identification has been assessed in previous work in a Remote Sensing and Geoinformation related Journal publication that we didn't address in detail due to anonymity concerns.
In this work want to present these more methodically-relevant evaluations for a more broader audience working with spatiotemporal data in general and hope that our results can be mirrored by other research fields.

Regarding your out-of-interest remark. In fact, we did train the ConvLSTM bi-directionally. The LSTM weights between forward- and backward passes have been shared which introduced no additional weights to the models.
We would actually argue that bi-directional encoding is also important in remote sensing to avoid a bias towards later observations, which motivated this design decision.
The results in Figure 2 show activations of the in-sequence pass, while similar activation maps have been created using the backward pass.
Since it has raised confusion, we will provide further details in a final version if this work.

Thank you for your perspective on the perception of your work. 

Best,

The Authors
title: To address your concerns and the methodical soundness of this work

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/AnonReviewer2
relevance: 3: borderline
impact: 2: low
novelty: 3: borderline

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper16/AnonReviewer2
evaluation: - The network architecture is architecture is a bit hard to understand, sentences like " this hyper-parameter of
75 dimensionality for the hidden tensors" are hard to follow. Maybe a diagram illustrating the architecture would help to better understand the overall picture.
- The choices of hyper-parameters are quite vague, why not use traditional CNN's architectures for feature extraction and then feed them into LSTM for the temporal classification?
-Results could be more detailed Figure 3 mentions results over training dataset only.
- I am not convinced that the work presented is better than any of the compared methods (e.g.hand crafted processing pipelines) as the results comparison seems to be missing appropriate analysis.

From: ~Frederik_Kratzert1
comment: I like the idea of this study, as well as this topic but I have some questions/remarks.

- Figure 3 and Results in general: It seems like you either train your network with all data or only present training results. This is rather unusual and I'm e.g. not sure if you can make any conclusion by comparing training accuracy over time. Especially statements like "X is performing better/worse than Y". Also, because your training samples are different for the problems (maybe not the samples, but the sequence length in the sample, which is kind of the same because your network sees less data to update the weights).

- What is the result of the presented study is unclear to me. You state that the preliminary task was to classify pixels into type of crops but you never present any classification results. This refers back to my last question: If you want to show, that your approach performs good and what the contribution of cloud cover in the data is, you should have a training and separate validation and test set. Only then you could honestly compare the different experiments you made.

- Out of interest: Did you run experiments with ConvLSTMs without the bidirectional encoding? This is used in NLP to encode an input sequence, because e.g. for translation the entire sentence is important and the sequence in the input might not be important for the target language. However, in this domain I think sequential order could be important since different crops have different states of growth over time.

minor comments:
- line 32: "fmaskalgorithm" -> "fmask algorithm"
- line 36: Do you mean deep neural networks by "deep methods" (or neural networks in general?). In my opinion "deep methods" is not a correct term.
- line 43: If [7] needs pre-classified cloud labels as inputs I think it needs the additional step of pre-classification, no? The two sentences (line 42ff) sound controversial.
- line 53 "map" -> "maps"
- Figure 2: Is the image for j and i correct at t_26? Is there not a single pixel where the input gate is open? Same for t_1
- Why do you think, that the clouds at t_34 are not detected by the input gate?
- line 93 "particolar" -> "particular"
- line 108: I'm not too familiar with remote sensing but statements like "as being common practice in XYZ" usually need a reference to validate the statement.
- line 116 "Datasets containing observations..." I think here is something missing (maybe "all"?).

title: Some comments to the presented study