Paper:
Published as a conference paper at ICLR 2018 FRATERNAL DROPOUT
Recurrent neural networks (RNNs) form an important class of architectures among neural networks useful for language modeling and sequential prediction. However, optimizing RNNs is known to be harder compared to feed-forward neural networks. A number of techniques have been proposed in literature to address this problem. In this paper we propose a simple technique called fraternal dropout that takes advantage of dropout to achieve this goal. Specifically, we propose to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions. In this way our regularization encourages the representations of RNNs to be invariant to dropout mask, thus being robust. We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout. We evaluate our model and achieve state-of-the-art results in sequence modeling tasks on two benchmark datasets -Penn Treebank and Wikitext-2. We also show that our approach leads to performance improvement by a significant margin in image captioning (Microsoft COCO) and semi-supervised (CIFAR-10) tasks.

Review:
The authors present Fraternal dropout as an improvement over Expectation-linear dropout (ELD) in terms of convergence and demonstrate the utility of Fraternal dropout on a number of tasks and datasets.
At test time, more often than not, people apply dropout in deterministic mode while at training time masks are sampled randomly. The paper addresses this issue by trying to reduce the gap.
I have 1.5 high level comments:
- Dropout can be applied by averaging results corresponding to randomly sampled masks ('MC eval'). This should not be ignored, and preferrably included in the evaluation.
- It could be made clearer why the proposed regularization would make the aforementioned gap smaller. Intuitively, the bias of the deterministic approximation (compared to the MC eval) should also play a role. It may be worth asking whether the bias changes? A possibility is that MC and deterministic evaluations meet halfway and with fraternal dropout MC eval is worse than without.
Details:
- The notation is confusing: p() looks like a probability distribution, z looks like a latent variable, p^t and l^t have superscripts instead of Y having a subscript, z^t is a function of X. Wouldn't f(X_t) be preferrable to p^t(z_t)?
- The experiments are set up and executed with care, but section 4 could be improved by providings details (as much as in section 5). The results on PTB and Wikitext-2 are really good. However, why not compare to ELD here? Section 5 leads the reader to believe that ELD would be equally good.
- Section 5 could be the most interesting part of the paper. This is where different regularization methods are compared (by the way, this is not "ablation"). It is somewhat unfortunate that due to lack of computational resources the comparisons are made at a single hyperparameter setting.
All in all, the results of section 4 are clearly good, but are they better than those of ELD? Evaluation and interpretation of results in section 5 is made difficult by the omission of the most informative quantity which Fraternal dropout is supposed to be approximating.