Paper:
Current state-of-the-art results in multilingual natural language inference (NLI) are based on tuning XLM (a pre-trained polyglot language model) separately for each language involved, resulting in multiple models. We reach significantly higher NLI results with a single model for all languages via multilingual tuning. Furthermore, we introduce cross-lingual knowledge distillation (XD), where the same polyglot model is used both as teacher and student across languages to improve its sentence representations without using the end-task labels. When used alone, XD beats multilingual tuning for some languages and the combination of them both results in a new state-of-the-art of 79.2% on the XNLI dataset, surpassing the previous result by absolute 2.5%. The models and code for reproducing our experiments will be made publicly available after de-anonymization.

Review:
This paper proposes two improved strategies for fine-tuning XLM (a multilingual variant of BERT) for cross-lingual NLI. First of all, it shows that fine-tuning a single model on the combination of all languages (the original English data from MultiNLI and their MT translation into the rest of languages) performs better than fine-tuning a separate model for each language. Furthermore, they show that minimizing the L2 distance between the English training sentences and their MT translation into the rest of languages, which does not explicitly use any labels in the foreign languages and is presented as a way of performing cross-lingual knowledge distillation, also performs better than zero-shot transferring a regular model fine-tuned in English.
I think that the paper makes some interesting contributions and, in particular, I think that the finding that multilingual fine-tuning performs better than the standard approach of fine-tuning a separate model for each language is important. Nevertheless, I am not convinced that there is enough novelty and substance on this, I have some concerns on the evaluation, and I think that the overall presentation should also be improved:
- I am not convinced by the "knowledge distillation" approach. First, although I see the connection, I do not think that presenting this as "knowledge distillation" is consistent with the common use of this term in the literature. More importantly, I do not see what is the value of this approach considering that multilingual fine-tuning performs better, and combining them both does not bring any clear improvement. The authors motivate it as a form of performing zero-shot cross-lingual transfer as, unlike the multilingual fine-tuning, this approach does not use any label in the foreign languages. However, I am not convinced at all by this reasoning, as it still relies on the translation of the English labeled data into the other languages. So, from a practical perspective, it requires the exact same resources as the other approach, as you would always be able to use the English labels for the rest of the languages, while being more complex and worse.
- It looks like the IndT results, which is the real baseline, are taken from the original XLM paper, while the rest of the results come from the authors' own runs, who use a different implementation. I think that you should also report IndT results from your own runs to make sure that your improvements come from the actual method, and not from small implementation details.
- You are trying small variations of your method (e.g. removing a particular language from the multilingual training) to support your claims, and it is not clear if the (rather small) differences in the results are significant. It would be good if you at least run the baseline multiple times and show the variance.
- It is unfair to try so many variants of your method in the test set, and then pick the best and claim SOTA as done in Table 6. Your final system looks rather ad-hoc and arbitrary: it is doing multilingual fine-tuning in all languages but Urdu, and cross-lingual knowledge distillation in a subset of 4 languages out of 15. It might get SOTA results in this particular scenario, but what if we move to a different set of languages, a different task, or even a different test set?
- The authors claim that "Urdu (ur) is an unrelated language" and "Swahili (sw) is loosely between French and Urdu in terms of relatedness to English", which they use to justify why Urdu behaves differently in their experiments. I do not speak neither Swahili nor Urdu, but from what I know this statement looks wrong. Swahili and English belong to completely different language families, and from what I know their grammar is very different. In contrast, Urdu at least belongs to the Indoeuropean language family.
- This is not relevant at all, but I would suggest the authors to find a different acronym instead of XD, which happens to be a widely used emoticon. I assume that the authors deliberately made this choice thinking that it would be funny, but I just find it confusing to see XD all over the place in a formal paper.