Paper:
MULTILINGUAL NEURAL MACHINE TRANSLATION WITH KNOWLEDGE DISTILLATION
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

Review:
Summary: Train a multilingual NMT system using the technique of Johnson et al (2017), but augment the standard cross-entropy loss with a distillation component based on individual (single-language-pair) teacher models. Periodically compare the validation BLEU score of the multilingual model with that of each individual model, and turn off distillation for language pairs where the multilingual model is better. On three different corpora (IWSLT, WMT, TED) with into-English translation from numbers of source languages ranging from 6 (WMT) to 44 (TED), this technique outperforms standard distillation for every language pair, and outperforms the individual models for most language pairs. Supplementary experiments justify the strategy of selectively turning off distillation, and quantify the effect using only the top 8 vocabulary items in distillation.
The main idea makes sense, and the results are very convincing, especially since it appears that hyper-parameters were not tuned extensively (eg, weight of 0.5 on the distillation loss, for all language pairs). Implementation should be very straightforward, especially with the trick of pre-computing top-k probabilities from the teacher model at each corpus position. One small barrier to practical application that the authors fail to acknowledge is the requirement to train individual models, which will at least double training time compared to a single multilingual model.
The main missing experiment is higher-capacity multilingual models, which Johnson et al show to be beneficial in settings with a large number of language pairs. Using a multilingual model of the same (relatively small) size as the individual models as is done here is likely to be suboptimal, especially for the 44-language pair TED setting. A related point is that the corpora used seem to be quite small (eg 4.5M and 1M sentences for WMT Czech and German, respectively, while the available training corpora are closer to 15M and 4.5M). Although performance relative to individual models is still impressive - and seems to be better than than in previous work - this makes the experiments comparing to the multilingual baseline less meaningful.
Also missing are experiments on out-of-English translation, which would establish the viability of the proposed technique for many-to-many translation via bridging. Out-of-English is a more difficult problem than into-English. I can’t see any reason the proposed technique wouldn’t also work in this setting, but this remains to be shown.
Although it’s great that the technique is shown to work without embellishments, there are a few obvious strategies it would have been interesting to explore, such as making the weight on the distillation loss dependent on the difference in performance between the multilingual and individual models; and allowing for the distillation loss to be turned back on if the performance of the multilingual model starts to drift back down for a particular language pair. I also wondered about the effect of the gradient accumulation strategy in algorithm 1, where individual batches from each language pair are effectively grouped into one giant batch for the purpose of parameter updates. I can see that this could stabilize training, but it would be good to know whether it’s crucial for success, especially when the number of language pairs is large.
Further details:
As aforementioned -> As mentioned
(1) 2nd line: Doesn't make sense as written. You need to distinguish the gold
y_t from hypothesized ones in the 1() function.
Above (2): is served as -> serves as
3.2 First paragraph. Since D presumably consists of D^l for all languages l,
L_ALL(D,...) should be a function of teacher parameters theta^l for all
languages l rather than just one as written.
In top-K distillation, is the teacher distribution renormalized or simply
truncated?
Generalization analysis, pg 8: presumably you are sampling from N(0, sigma^2) -
this should be described as such.
Reference: 
Johnson et al, “Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation” TACL, 2017.