Paper:
Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub-networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT-like models of higher quality compared to training from scratch or using distillation.

Review:
This paper presents LayerDrop, a simple method for dropping groups of weights (typically layers) jointly. Despite its simplicity (which is actually a big plus), the method seems to improve performance quite consistently on a range of NLP tasks. Moreover, it allows the authors to train very deep networks, that are very hard to train otherwise (according to the authors). For me the most exciting thing about this approach is that this training regime allows to prune the trained network at test time *without finetuning*, effectively getting a smaller, more efficient network for free. This is a great benefit compared to existing approaches that require retraining a smaller network for each costume size. While the method isn't really applicable to any size, and largely depends on the dropout rate the full model was trained on, I imagine it could serve as a starting point for other researchers to develop more flexible extensions that would allow for any size of network to be pruned at test time. I think this is a very strong submission and strongly advocate accepting it to ICLR.
Questions and comments:
1. The main thing missing for me is some more analysis on the runtime/energetic savings (e.g., in terms of FLOPs) of the proposed method. The authors argue (3.2.1) that approaches such as DropConnect are not necessarily more efficient, but do not analyze the efficiency of their pruned networks apart from the size of the pruned network. 
2. Similarly, details about the experiments are also somewhat lacking:
a. how many GPUs were used to train the models? the authors mention 8 v100 in A.3, but I am not sure if this was the setup for all experiments. 
b. Figure 7, which shows that LayerDrop also improves training speed, is very interesting and should be part of the main text in my opinion. Was this trend consistent for all experiments?
c. Similarly, presenting the total running time of the models (and not just words per second) would be helpful for reproducibility. 
d. Finally, reporting dev and not only test results (e.g., in tables 1 and 2) would also facilitate future reproducibility efforts.
3. Did the authors use a regular dropout? If I understand correctly, in A.1.3, the authors mention tuning the dropout rate between {0.2,0.3}. Was this done for all tasks? and was it done for the baseline models as well? Using dropout in the baseline model with a similar proportion as LayerDrop seems like an important baseline, and in particular it would be interesting to see whether the deep experiments (e.g., 40 layers on WT103) that are hard to train without LayerDrop could converge with regular dropout.
Minor:
- 3.2: "We present *an* regularization approach ..." (should be "a")
- Table 2 is referred to before table 1, it might be clearer to switch them.
- In figure 4, it wasn't clear to me why "Layer" on the lefthand side is much better than "Every other" on the righthand side. Aren't these the same model variant?
- Missing year for paper "Language models are unsupervised multitask learners".