Paper:
Predicting Scene Parsing and Motion Dynamics in the Future
The ability of predicting the future is important for intelligent systems, e.g. autonomous vehicles and robots to plan early and make decisions accordingly. Future scene parsing and optical flow estimation are two key tasks that help agents better understand their environments as the former provides dense semantic information, i.e. what objects will be present and where they will appear, while the latter provides dense motion information, i.e. how the objects will move. In this paper, we propose a novel model to simultaneously predict scene parsing and optical flow in unobserved future video frames. To our best knowledge, this is the first attempt in jointly predicting scene parsing and motion dynamics. In particular, scene parsing enables structured motion prediction by decomposing optical flow into different groups while optical flow estimation brings reliable pixel-wise correspondence to scene parsing. By exploiting this mutually beneficial relationship, our model shows significantly better parsing and motion prediction results when compared to well-established baselines and individual prediction models on the large-scale Cityscapes dataset. In addition, we also demonstrate that our model can be used to predict the steering angle of the vehicles, which further verifies the ability of our model to learn latent representations of scene dynamics.

Review:
The paper proposes a deep-learning-based approach to joint prediction of future optical flow and semantic segmentation in videos. The authors evaluate the approach in a driving scenario and show that the two components - flow prediction and semantic segmentation prediction - benefit from each other. moreover, they show a proof-of-concept experiment on using the predictions for steering angle prediction.
The paper is related to works of Jin et al. and Neverova et al. However, as far as I understand, both of these have not been officially published at the time of submission (and the work of Neverova et al. only appeared on arxiv 2 months before the deadline), so they should not be considered prior art.
Detailed comment:
Pros:
1) The idea seems sound: predicting segmentation and optical flow are both important tasks, and they should be mutually beneficial.
2) The architecture is clean and seems reasonable
3) Experimental evaluation on Citiscapes is reasonable, and some relevant baselines are evaluated: copying previous frame, warping with the estimated flow, predicting segmentation without flow, predicting flow without segmentation, the proposed model without the “transformation layer”. The improvements brought by the modifications are not always huge, but are at least noticeable.
Cons:
1) Steering angle prediction experiments are not convincing at all. The baseline is very unclear, and it is unclear what does the performance of “~4” mean. There is no obvious baseline - training a network with the same architecture as used by the authors, but trained from scratch. The idea of the section is good, but currently it’s not satisfactory at all.
2) I am confused by the loss function design. First, the weighting of the loss component is not mentioned at all - is there indeed no weighting? Second, the semantic segmentation loss is different for human-annotated and non-human-annotated frames - why is that? I understand why human-annotated loss should have a higher weight, but why use a different function - L1+GDL instead of cross-entropy? Have you experimented with other variants?
3) Figures 3 and 4 are not particularly enlightening. Honestly, they do not tell too much. Of course it is always possible to find a sample on which the proposed method works better. The authors could find a better way to represent the results - perhaps with a video, or with better-designed figures.
4) Writing looks rushed and unpolished.  For instance, caption of Table 2 is wrong.
Minor:
5) I’m somewhat surprised the authors use EpicFlow - it’s not quite state of the art nowadays
6) Missing relevant citation: M. Bai, W. Luo, K. Kundu and R. Urtasun: Exploiting Semantic Information and Deep Matching for Optical Flow. ECCV 2016.
7) Many officially published papers are cited as arxiv, to mention a few: Cityscapes, FlowNet, ResNet, Deep multi-scale video prediction, Context encoders
Overall, the paper is interesting and novel, but seems somewhat rushed, especially in some sections. Still, I believe it could be accepted if the authors fix some most embarrassing problems mentioned above, in particular, the part on steering angle prediction.