Paper:
Published as a conference paper at ICLR 2018 NEURAL SPEED READING VIA SKIM-RNN
Inspired by the principles of speed reading, we introduce Skim-RNN, a recurrent neural network (RNN) that dynamically decides to update only a small fraction of the hidden state for relatively unimportant input tokens. Skim-RNN gives computational advantage over an RNN that always updates the entire hidden state. Skim-RNN uses the same input and output interfaces as a standard RNN and can be easily used instead of RNNs in existing models. In our experiments, we show that Skim-RNN can achieve significantly reduced computational cost without losing accuracy compared to standard RNNs across five different natural language tasks. In addition, we demonstrate that the trade-off between accuracy and speed of Skim-RNN can be dynamically controlled during inference time in a stable manner. Our analysis also shows that Skim-RNN running on a single CPU offers lower latency compared to standard RNNs on GPUs.

Review:
Summary: The paper proposes a learnable skimming mechanism for RNN. The model decides whether to send the word to a larger heavy-weight RNN or a light-weight RNN. The heavy-weight and the light-weight RNN each controls a portion of the hidden state. The paper finds that with the proposed skimming method, they achieve a significant reduction in terms of FLOPS. Although it doesn’t contribute to much speedup on modern GPU hardware, there is a good speedup on CPU, and it is more power efficient.
Contribution:
- The paper proposes to use a small RNN to read unimportant text. Unlike (Yu et al., 2017), which skips the text, here the model decides between small and large RNN.
Pros:
- Models that dynamically decide the amount of computation make intuitive sense and are of general interests.
- The paper presents solid experimentation on various text classification and question answering datasets.
- The proposed method has shown reasonable reduction in FLOPS and CPU speedup with no significant accuracy degradation (increase in accuracy in some tasks).
- The paper is well written, and the presentation is good.
Cons:
- Each model component is not novel. The authors propose to use Gumbel softmax, but does compare other gradient estimators. It would be good to use REINFORCE to do a fair comparison with (Yu et al., 2017 ) to see the benefit of using small RNN.
- The authors report that training from scratch results in unstable skim rate, while Half pretrain seems to always work better than fully pretrained ones. This makes the success of training a bit adhoc, as one need to actively tune the number of pretraining steps.
- Although there is difference from (Yu et al., 2017), the contribution of this paper is still incremental.
Questions:
- Although it is out of the scope for this paper to achieve GPU level speedup, I am curious to know some numbers on GPU speedup.
- One recommended task would probably be text summarization, in which the attended text can contribute to the output of the summary.
Conclusion:
- Based on the comments above, I recommend Accept