Paper:
This paper proposes a novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during decoding. In sequenceto-sequence modeling, typically, the output of the last layer of the N -layer encoder is fed to the M -layer decoder, and the output of the last decoder layer is used to compute loss. Instead, our method computes a single loss consisting of N × M losses, where each loss is computed from the output of one of the M decoder layers connected to one of the N encoder layers. A single model trained by our method subsumes multiple models with different number of encoder and decoder layers, and can be used for decoding with fewer than the maximum number of encoder and decoder layers. We then propose a mechanism to choose a priori the number of encoder and decoder layers for faster decoding, and also explore recurrent stacking of layers and knowledge distillation to enable further parameter reduction. In a case study of neural machine translation, we present a cost-benefit analysis of the proposed approaches and empirically show that they greatly reduce decoding costs while preserving translation quality.

Review:
This work proposes a way to reduce the latencies incurred in inference for neural machine translation. Basic idea is to train a model with softmax attached to each output of decoder layers, and computes a loss by aggregating the cross entropy losses over the softmaxes. During inference, it could either use one of the softmax or train an additional model which dynamically selects softmaxes given an input string. Experimental results show that it is possible to reduce latencies by trading off the translation qualities measured by BLEU. Dynamically selection did not show any gains in latencies, though, this work empirically shows potential gains in oracle studies. This work further shows that the model could be compressed further by knowledge distillation.
I have several concerns to this work and I'd recommend rejecting this submission.
- One of the problems of this paper is presentation. This work basically combines three work together as a single paper, i.e., section 3 for the basic model, section 4 for dynamic selection and section 5 for distillation, with each section describing a separate experiment. I'd strongly suggest the author to focus on the main point, e.g., dynamic selection, and present the basic model and dynamic selection. Experiments should be presented in a single section for brevity.
- Similarly, this work should have been submitted when meaningful gains were observed in the dynamic selection method, given that the proposal is somewhat new. Otherwise, I don't find any merits to see this accepted in ICLR, given the rather negative results in section 4.
- The description in section 4.2 is totally messed up. x^i and y^i_k are strings since they are an input sentence and an output sentence, respectively,. However, they are treated as scalars in Equation 3 by multiplied with \delta_k, subtracted from 1 and taking sigmoid through \sigma. I strongly suggest authors to carefully check variables used in the equations and the description in the section.
- The authors claim that the use of knowledge distillation is novel. However, it is already widely known in the research community and I don't think it is worthy to keep it as a single section. It could have been described as a yet another experiment in a single experimental section.
Other comment:
- Although this paper claims that attaching a softmax for each output layer is new, there is a similar work in language modeling, though the motivation is totally different.
  Direct Output Connection for a High-Rank Language Model, Sho Takase, Jun Suzuki and Masaaki Nagata, EMNLP 2019.
- In section 3.4, this paper claims that the training of all 36 models took 25.5 more time, but took 9.5 more time for a tied-model when compared with a basic 6-layer Transformer. It is not clear to me whether this comparison is meaningful given that it might be possible to employ multiple machines to train 36 models.