Paper:
Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model
With the goal of making high-resolution forecasts of regional rainfall, precipitation nowcasting has become an important and fundamental technology underlying various public services ranging from rainstorm warnings to flight safety. Recently, the Convolutional LSTM (ConvLSTM) model has been shown to outperform traditional optical flow based methods for precipitation nowcasting, suggesting that deep learning models have a huge potential for solving the problem. However, the convolutional recurrence structure in ConvLSTM-based models is location-invariant while natural motion and transformation (e.g., rotation) are location-variant in general. Furthermore, since deep-learning-based precipitation nowcasting is a newly emerging area, clear evaluation protocols have not yet been established. To address these problems, we propose both a new model and a benchmark for precipitation nowcasting. Specifically, we go beyond ConvLSTM and propose the Trajectory GRU (TrajGRU) model that can actively learn the location-variant structure for recurrent connections. Besides, we provide a benchmark that includes a real-world large-scale dataset from the Hong Kong Observatory, a new training loss, and a comprehensive evaluation protocol to facilitate future research and gauge the state of the art.

Review:
Summary
This paper describes a new GRU-based architecture for precipitation nowcasting, a task which can be seen as a video prediction problem with a fixed camera position. The authors also describe a new benchmarking package for precipitation nowcasting and evaluate their model on both this benchmark and an altered version of moving MNIST.
Technical quality
I think this is a good application paper while the proposed architecture is interesting as well. For the sake of completeness I would have liked to see some comparisons with LSTM versions of the networks as well and with fully connected RNNs but the authors already did look at a nice selection of baselines. 
Clarity 
The paper is generally well-written and coherent in structure. While I found it easy to follow the definition of the ConvGRU, I found the description of the proposed TrajGRU much harder to understand. The most important change in the new model is the replacement of the state-to-state convolutions/multiplications with a transformation that seems to be a specific version of the type of module described by Jaderberg et al. (2015) (reference 10 in the paper). It is not clear to me how the structure generating network gamma produces the flow fields U and V exactly though. It would help if this was made more explicit and if there would be a more clear emphasis on the differences with the transformation module from Jaderberg et al. (2015). At least the goal of the trajectory selection mechanism itself is clear enough for the rest of the narrative to make sense. 
Novelty
While the general architecture is the same as the ConvLSTM and the paper is somewhat application oriented, the idea of learning trajectories with spatial transformer inspired modules is quite original to me.
Significance
The new data set seems to be a useful benchmark but I know too little about precipitation nowcasting to judge this. I also don't know enough about the state-of-the-art in that field to judge the performance gains very well and since the benchmark is a new one I suppose time will need to tell. The work would have been stronger if there was also a comparison on another existing precipitation nowcasting benchmark. I think that the idea to learn sparsely connected temporal trajectories may have more widespread applications in other domains where the distributions to be modelled are of high dimensionality but display certain independence properties and redundance. 
pros:
Interesting model
Nice baselines and practical evaluation
The paper is generally well-written.
cons:
The description and motivation of the model is a bit short and hard to follow.
The novelty is somewhat limited compared to ConvGRU/LSTM
EDIT: The rebuttal addressed some of the concerns I had about the paper and I think that the additional experiments also add to the overall quality so I increased my score.