From: ~Nelson_Fernandez1
comment: Dear Ransalu,

That e-mail address is not longer available, my current account is nelson.fernandez-pinto@renault.com

Thanks,

Nelson

title: Registration

From: ~Nelson_Fernandez1
comment: Hello, thank you for your constructive comments,

We have some evaluations of MTL's contribution that were not included due to limited space, specifically the ablation study. We found that the auxiliary task did not have a significant impact on RMSE but only on the whiteness (smoothness of prediction in time). This is caused by the inductive bias effect described by Caruana (1997). We could add them in a next (quick) revision if necessary.
title: About the official comment

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper4/AnonReviewer2
evaluation: The official comment (AnonReviewer2) still carries my view on this paper. Overall it is a reasonable initial investigation of augmenting the conventional steering based steering prediction techniques with optical flow via the two-stream architecture. However, possibly due to the limited space, there is little analysis on the effect of the auxiliary MTL task and would be nice to see more discussion and possibly ablation study. 

For a workshop paper, I'm leaning towards the positive side of borderline but would not be upset if the paper was rejected due to a large number and high quality of other submissions.

With the points I made in the official comment for improving the paper, this could be a solid submission to a conference with further analysis and evaluation.

From: ~Nelson_Fernandez1
comment: Hello, thank you very much for the evaluation and comments, here some aclaratories:

1.- The prediction of the previous steering angle was motivated exclusively by MTL. The main advantage we found is a first-degree filtering effect caused by induction of bias during training. In other words, when trying to predict simultaneously the current and previous steering angle, the bias of the network is modified, converging towards a solution that is optimal for both tasks [1] gaining prediction stability in the temporal domain. This variable is an auxiliary variable, its precise study is beyond the scope of the work presented. 

2.- Several (recent) works are being published using the object displacement representation for end-to-end learning, please see [2] and [3]. There is also a growing interest in the use of biologically inspired event-based cameras that only encode pixel variation over time [3] [4]. As you said, there is an inherent relationship between the vehicle motion (caused by driver’s actions) and the motion field. This does not mean the representation is not useful for improving the steering angle prediction task. In fact, the significant reduction in RMSE achieved implies greater generalization capabilities that cannot be explained by the CNN architecture used. I agree with you on the fact that motion field representation to decision making of an autonomous agent should be studied in depth in future works.

The advantage of the proposed architecture is the combination of two representations, the raw images, that capture essentially spatial cues [5] and optical flows, encoding object displacement and temporal dynamics. The network has been trained to take advantage of both embedded representations, so it is an incremental step, relative to the use raw images only (a system that actually works). 

The next steps you propose sound very interesting, we will go further to perform simulator tests. Thank you very much for your review.

[1] Caruana, R. (1997). Multitask learning. Machine learning, 28(1), 41-75.
[2] Lex Fridman and Li Ding and Benedikt Jenik and Bryan Reimer. (2017) Arguing Machines: Human Supervision of Black Box AI Systems That Make Life-Critical Decisions. arXiv 1710.04459.
[3] Maqueda, A. I., Loquercio, A., Gallego, G., Garcıa, N., & Scaramuzza, D. (2018, April). Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5419-5427).
[4] Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., & Benosman, R. (2018, March). HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1731-1740).
[5] Bojarski, M., Yeres, P., Choromanska, A., Choromanski, K., Firner, B., Jackel, L., & Muller, U. (2017). Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911.

title: Comments

From: ~Ransalu_Senanayake1
comment: The paper investigates the use of the Two-Stream architecture which was originally proposed for action recognition in videos to the task of predicting the steering angle of a vehicle. Specifically, two CNNs are used for processing raw color images and pre-processed optical flow respectively before fusing with a MLP. Additionally, the network architecture predicts the steering for both the current frame and the previous frame as an auxiliary task motivated by MTL. The results demonstrate the beneficial impact of including optical flow with a significant reduction in RMSE. However, the addition of predicting the previous steering angle is not assessed and left for future work.

The main issue I have with this work is that by the nature of the problem, the optical flow is directly influenced by the turn rate of the vehicle so the findings are not surprising. Further evaluation should include a baseline model which works purely off the mean flow vector to better assess the value of a complete flow field. 
Moreover, as the optical flow is caused by the action taken by the vehicle/driver (more so than being a cue for decision making) it is unclear how this would benefit self-driving cars. Future work should include evaluation in a simulator where the output of the network steers an agent which has a causal effect on the sequence of inputs.
title: From: NIPS 2018 Workshop Spatiotemporal Paper4 AnonReviewer2 (appeared as an official comment)

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper4/AnonReviewer2
relevance: 4: high
impact: 2: low
novelty: 3: borderline

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper4/Authors
comment: Hello, thank you very much for your evaluation/comments. Here are some clarifications:

Major comments
- The citation will be added to the next revision.

- Here we present another argument that motivates the fully convolutional approach we selected. This assertion is based on the assumption that the raw images and optical flows provide all the information necessary to infer the current and previous state of the target variable. Therefore, there is no explicit need to use recurrent layers to `remember' the previous steering angle. This phrase is not crucial to the concept and could be simply dropped at the next revision.

- A lower batch size exploits the stochasticity of the gradient calculation during backpropagation. It induces an interesting generalization effect during the last part of the training.

- An intuitive explanation of the equation will be added to the next revision.

- We argue that 2D-CNN trained with optical flows was biased to learn features only as spatial representations (as if it were a clean road lane segmentation, for example). As we add more sophistication to the architecture (two streams and multitask learning) an important bias correction is achieved in the temporal domain, reducing the whiteness. Additional comments on this topic will be added to the next revision.

Minor comments
- The number of test samples will be added to the next revision.
- The sentence will be corrected on the next revision.

Best regards
title: Clarifications

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper4/AnonReviewer1
evaluation: This paper proposes an extension to the two-stream CNN proposed in Symonyan and Zisserman (2014) for regression and demonstrates its potential in a steering angle prediction task. A two stream CNN learns from two flows of information. Spatial information is gathered from raw images, where temporal information is learnt from an optical flow.

The training consists of two phases; a training phase which trains the two streams independently, and a finetuning phase, which trains the MLP regression layer. The experiments show that there is a clear advantage of using a two-stream CNN model, compared to using just raw images or optical-flow only.

Overall, I believe this paper demonstrates interesting findings on the use of potential usage of two-stream networks, to leverage spatio-temporal information to develop better autopilot systems. The experiments are well designed to show the benefits both quantitatively and qualitatively. There are some concerns about several statements and details of the experiments as outlined below.


Strengths:
------------------------
+ The motivation for using two stream network is well delivered

+ Sound experiments showing the benefit of two-stream approach compared to other raw image based, and optical-flow based benchmarks.



Major comments
------------------------
Please include citations for the statement "2D-CNNs are less likely to overfit when performing temporal tasks" (line 31).

Line 62-63: It is not clear why it makes no practical sense to use RNNs. Please clarify.

I can see the benefit of having a lower learning rate at the finetuning phase compared to training phase, but what is the reasoning behind using a smaller batch size for the finetuning phase?

At least a brief explanation about the notation in Equation 1 is required in the text. Otherwise, it seems very disconnected from text.

The paper discusses that raw images contribute to learning spatial information, where optical flow enables the network to learn the temporal information. Given this statement, I expected a lower Whiteness value for the networks trained with optical flow in Table 1. However, it is not the case. It would be good if authors can comment on this observation.


Minor comments
------------------------
Please mention the number of images contained in the test set (similar to of the training set).

Line 75: "... weights obtained are then used as initialization." -> "... weights obtained are then used as the initializaiton for the second stage."

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper4/AnonReviewer1
relevance: 5: very high
impact: 4: high
novelty: 3: borderline

From: ~Oliver_Paul_Richter1
comment: As self-driving cars is not my field of expertise I have no further questions for now :)
title: Thanks for the clarifications

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper4/Authors
comment: Hello, thank you very much for your constructive comments. Here are some answers:

1.- You may be right about the terminology, we will adjust the title so that it is more focused on end-to-end learning, good observation!

2.- To establish the current SOTA, we used the same references given by Eraqi [5]. However, we could obviate the use of that word to avoid misunderstandings.

3.- We are confident about the baseline results. All regression models were fairly trained. They were also tested and implemented in a 1:10 RC car. We made an exhaustive work finding the best hyperparameters, and we applied techniques to avoid overfitting (data augmentation and dropout). The baseline results are easily reproducible and are not far from what was expected, there are no surprises here. 

3.- The training process seems to be a little over engineered, as you said. However, it is based on the concept of "hard parameters sharing" of Caruana's work [6]. It is very useful to have pre-trained CNN streams before training the MTL module. It also allowed us to quantify with greater precision the contribution of MTL to the prediction stability.

4.- We have tested several possible configurations and the one presented has the best performance. It is important to note that the architecture presented is largely inspired by Simonyan's work on action recognition in videos [4] (as indicated in the abstract). The main idea of this work was to extend the same concept to see if it could improve the other regression methods as it does.

5.- Yes, we have some experiments underway, we are planning to add the results as a revision.

Thank you again for your comments, surely it helps.

title: Some answers

From: ~Oliver_Paul_Richter1
comment: Hi, thanks for the paper! Some comments/questions from my side:
 - Steering angle prediction is not autonomous steering: In the paper you present an architecture that predicts the steering angle and improves upon other architectures. This stands in contrast to the title and abstract where you advertise "autonomous steering" and "end-to-end learning for self-driving cars". Although the two fields are related, autonomous steering is a much harder task due to the feedback loop. I would not try to oversell the paper and focus on what you are actually doing: steering angle prediction. Additionally you could then give an intuition why steering angle prediction is an important task to develop a self driving system.
 - The results seem unclear: You state that you achieve "competitive improvement of 30% in prediction accuracy and stability compared to state-of-the-art methods". However, I couldn't find in the paper what these "state-of-the-art" (SOTA) methods are. Further, are these SOTA in image classification or steering angle estimation? Who claimed that these are SOTA? In your evaluation you show that your method achieves better results than other architectures for the (over-engineered?) learning rate/training schedule you propose. However, it remains unclear to me whether you did a fair evaluation by tuning the hyperparameters to values that work good for all architectures tested.
 - The main improvement seems to come from using optical flows instead of RGB images (for the RMSE measurement) and from the additional task (for the Whiteness measurement). Have you also tried using only 1 CNN with optical flow input plus the two prediction tasks? Or have you tried frame stacking instead of optical flow calculation?
 - Even though you mention that RNNs take longer to train, a comparison in terms of accuracy would be nice.
 - There are some English mistakes in the paper, e.g., "[...] the implemented in [...]" (line 86 and footnote on page 3).
Hope this helps
title: Some comments/questions on the presented model and results