Paper:
When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is fluent in, even if the math lessons were only taught in one language. However, current representations in machine learning are language dependent. In this work, we present a method to decouple the language from the problem by learning language agnostic representations and therefore allowing training a model in one language and applying to a different one in a zero shot fashion. We learn these representations by taking inspiration from linguistics, specifically the Universal Grammar hypothesis and learn universal latent representations that are language agnostic (Chomsky, 2014;Montague, 1970). We demonstrate the capabilities of these representations by showing that models trained on a single language using language agnostic representations achieve very similar accuracies in other languages.

Review:
This paper starts with the bold aim of extracting Montague's universal grammar from multiple languages. In order to do so, the authors train multiple language models where each LM is explicitly factorized into language-specific and language-independent representations. The authors then apply the GAN framework to the language-independent parts to enforce all languages to share the same latent space. The claim here is that the language independent parameters capture the essence of universal grammar. The authors show that their framework enables effective zero-shot learning of tasks over new languages (for example sentiment classifier learned on top of English data generalize to Chinese when trained on the universal grammar embeddings). 
The paper is overall well written and the experimental results are convincing.
The gripe, however, I have is that this paper makes the claims that go too far without evaluating them. It is entirely sufficient to claim that you're trying to learn language agnostic parameters/embeddings --  I'd be happy with that. But the paper goes further and claims to be learning a form of universal grammar. To justify this claims, it is not sufficient to show in the experiments that the new representations do better at sentiment and NLI. The authors must show that this captures the "innate" language learning abilities akin to human babies. While the paper aims to do some analysis in the discussion section, it is not unsatisfactory. As the paper says in the discussion section "From a machine learning perspective, we’re interested in extracting informative features and not necessarily a completely grammatical language model. That being said it is of interest to what extent language models capture grammar and furthermore the extent to which models trained toward the universal grammar objective learn grammar."
The problem is that simply comparing LM perplexities is not a solid test of whether this model has learned some form of universal grammar. First, this paper does not define a clear falsifiable hypothesis on the proof of learning universal grammar. One example of testing for learning grammar can be: does this model learn basic syntactic rules of a new language (e.g. as the authors suggested -- head-first or head-final syntax, or rules of conjugation)  with a small amount of data after being training a universal representation with n languages? There have been a series of recent papers on checking if Language Models have appropriately learned syntax. See e.g. Tal Linzen's work https://arxiv.org/pdf/1809.04179.pdf. Just to be clear I am not suggesting citing works in unpublished places but potentially using some of the tests suggested in these papers.
In conclusion, I think this work is useful but I also think it makes really grandiloquent claims without verifying them. That to me is a dangerous precedent.