Paper:
Published as a conference paper at ICLR 2018 TWIN NETWORKS: MATCHING THE FUTURE FOR SEQUENCE GENERATION
We propose a simple technique for encouraging generative RNNs to plan ahead. We train a "backward" recurrent network to generate a given sequence in reverse order, and we encourage states of the forward model to predict cotemporal states of the backward model. The backward network is used only during training, and plays no role during sampling or inference. We hypothesize that our approach eases modeling of long-term dependencies by implicitly forcing the forward states to hold information about the longer-term future (as contained in the backward states). We show empirically that our approach achieves 9% relative improvement for a speech recognition task, and achieves significant improvement on a COCO caption generation task.

Review:
1) Summary
This paper proposes a recurrent neural network (RNN) training formulation for encouraging RNN the hidden representations to contain information useful for predicting future timesteps reliably. The authors propose to train a forward and backward RNN in parallel. The forward RNN predicts forward in time and the backward RNN predicts backwards in time. While the forward RNN is trained to predict the next timestep, its hidden representation is forced to be similar to the representation of the backward RNN in the same optimization step. In experiments, it is shown that the proposed method improves training speed in terms of number of training iterations, achieves 0.8 CIDEr points improvement over baselines using the proposed training, and also achieves improved performance for the task of speech recognition.
2) Pros:
+ Novel idea that makes sense for learning a more robust representation for predicting the future and prevent only local temporal correlations learned.
+ Informative analysis for clearly identifying the strengths of the proposed method and where it is failing to perform as expected.
+ Improved performance in speech recognition task.
+ The idea is clearly explained and well motivated.
3) Cons:
Image captioning experiment:
In the experimental section, there is an image captioning result in which the proposed method is used on top of two baselines. This experiment shows improvement over such baselines, however, the performance is still worse compared against baselines such as Lu et al, 2017 and Yao et al, 2016. It would be optimal if the authors can use their training method on such baselines and show improved performance, or explain why this cannot be done.
Unconditioned generation experiments:
In these experiments, sequential pixel-by-pixel MNIST generation is performed in which the proposed method did not help. Because of this, two conditioned set ups are performed: 1) 25% of pixels are given before generation, and 2) 75% of pixels are given before generation. The proposed method performs similar to the baseline in the 25% case, and better than the baseline in the 75% case. For completeness, and to come to a stronger conclusion on how much uncertainty really affects the proposed method, this experiment needs a case in which 50% of the pixels are given. Observing 25% of the pixels gives almost no information about the identity of the digit and it makes sense that it’s hard to encode the future, however, 50% of the pixels give a good idea of what the digit identity is. If the authors believe that the 50% case is not necessary, please feel free to explain why.
Additional comments:
The method is shown to converge faster compared to the baselines, however, it is possible that the baseline may finish training faster (the authors do acknowledge the additional computation needed in the backward RNN).
It would be informative for the research community to see the relationship of training time (how long it takes in hours) versus how fast it learns (iterations taken to learn).
Experiments on RL planning tasks would be interesting to see (Maybe on a simple/predictable environment).
4) Conclusion
The paper proposes a method for training RNN architectures to better model the future in its internal state supervised by another RNN modeling the future in reverse. Correctly modeling the future is very important for tasks that require making decisions of what to do in the future based on what we predict from the past. The proposed method presents a possible way of better modeling the future, however, some the results do not clearly back up the claim yet. The given score will improve if the authors are able to address the stated issues.
POST REBUTTAL RESPONSE:
The authors have addressed the comments on the MNIST experiments and show better results, however, as far as I can see, they did not address my concern about the comparisons on the image captioning experiment. In the image captioning experiment the authors choose two networks (Show & Tell and Soft attention) that they improve using the proposed method that end up performing similar to the second best baseline (Yao et al. 2016) based on Table 3 and their response. I requested for the authors to use their method on the best performing baselines (i.e. Yao et al. 2016 or Liu et al. 2017) or explain why this cannot be done (maybe my request was not clearly stated). Applying the proposed method on the strong baselines would highlight the author's claims more strongly than just applying on the average performing chosen baselines. This request was not addressed and instead the authors just improved the average performing baselines in Table 3 to meet the best baselines. Given, that the authors were able to improve the results in the sequential MNIST and improve the average baselines, my rating improves one point. However, I still have concerns about this method not being shown to improve the best methods presented in Table 3 which would give a more solid result. My rating changes to marginally above threshold for acceptance.