Paper:
Published as a conference paper at ICLR 2018 SKIP RNN: LEARNING TO SKIP STATE UPDATES IN RECURRENT NEURAL NETWORKS
Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc. github.io/skiprnn-2017-telecombcn/.

Review:
UPDATE: Following the author's response I've increased my score from 5 to 6. The revised paper includes many of the additional references that I suggested, and the author response clarified my confusion over the Charades experiments; their results are indeed close to state-of-the-art on Charades activity localization (slightly outperformed by [6]), which I had mistakenly confused with activity classification (from [5]).
The paper proposes the Skip RNN model which allows a recurrent network to selectively skip updating its hidden state for some inputs, leading to reduced computation at test-time. At each timestep the model emits an update probability; if this probability is over a threshold then the next input and state update will be skipped. The use of a straight-through estimator allows the model to be trained with standard backpropagation. The number of state updates that the model learns to use can be controlled with an auxiliary loss function. Experiments are performed on a variety of tasks, demonstrating that the Skip-RNN compares as well or better than baselines even when skipping nearly half its state updates.
Pros:
- Task of reducing computation by skipping inputs is interesting
- Model is novel and interesting
- Experiments on multiple tasks and datasets confirm the efficacy of the method
- Skipping behavior can be controlled via an auxiliary loss term
- Paper is clearly written
Cons:
- Missing comparison to prior work on sequential MNIST
- Low performance on Charades dataset, no comparison to prior work
- No comparison to prior work on IMDB Sentiment Analysis or UCF-101 activity classification
The task of reducing computation by skipping RNN inputs is interesting, and the proposed method is novel, interesting, and clearly explained. Experimental results across a variety of tasks are convincing; in all tasks the Skip-RNNs achieve their goal of performing as well or better than equivalent non-skipping variants. The use of an auxiliary loss to control the number of state updates is interesting; since it sometimes improves performance it appears to have some regularizing effect on the model in addition to controlling the trade-off between speed and accuracy.
However, where possible experiments should compare directly with prior published results on these tasks; none of the experiments from the main paper or supplementary material report any numbers from any other published work.
On permuted MNIST, Table 2 could include results from [1-4]. Of particular interest is [3], which reports 98.9% accuracy with a 100-unit LSTM initialized with orthogonal and identity weight matrices; this is significantly higher than all reported results for the sequential MNIST task.
For Charades, all reported results appear significantly lower than the baseline methods reported in [5] and [6] with no explanation. All methods work on “fc7 features from the RGB stream of a two-stream CNN provided by the organizers of the [Charades] challenge”, and the best-performing method (Skip GRU) achieves 9.02 mAP. This is significantly lower than the two-stream results from [5] (11.9 mAP and 14.3 mAP) and also lower than pretrained AlexNet features averaged over 30 frames and classified with a linear SVM, which [5] reports as achieving 11.3 mAP. I don’t expect to see state-of-the-art performance on Charades; the point of the experiment is to demonstrate that Skip-RNNs perform as well or better than their non-skipping counterparts, which it does. However I am surprised at the low absolute performance of all reported results, and would appreciate if the authors could help to clarify whether this is due to differences in experimental setup or something else.
In a similar vein, from the supplementary material, sentiment analysis on IMDB and action classification on UCF-101 are well-studied problems, but the authors do not compare with any previously published results on these tasks.
Though experiments may not show show state-of-the-art performance, I think that they still serve to demonstrate the utility of the Skip-RNN architecture when compared side-by-side with a similarly tuned non-skipping baseline. However I feel that the authors should include some discussion of other published results.
On the whole I believe that the task and method are interesting, and experiments convincingly demonstrate the utility of Skip-RNNs compared to the author’s own baselines. I will happily upgrade my rating of the paper if the authors can address my concerns over prior work in the experiments.
References
[1] Le et al, “A Simple Way to Initialize Recurrent Networks of Rectified Linear Units”, arXiv 2015
[2] Arjovsky et al, “Unitary Evolution Recurrent Neural Networks”, ICML 2016
[3] Cooijmans et al, “Recurrent Batch Normalization”, ICLR 2017
[4] Zhang et al, “Architectural Complexity Measures of Recurrent Neural Networks”, NIPS 2016
[5] Sigurdsson et al, “Hollywood in homes: Crowdsourcing data collection for activity understanding”, ECCV 2016
[6] Sigurdsson et al, “Asynchronous temporal fields for action recognition”, CVPR 2017