Paper:
Contextual representation models like BERT have achieved state-of-the-art performance on a diverse range of NLP tasks. We propose a cross-lingual contextual representation model that generates language-independent contextual representations. This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models, on top of contextual representation models like BERT. We provide a formulation of language-independent cross-lingual contextual representation based on mono-lingual representations. Our formulation takes three steps to align sequences of vectors: transform, extract, and reorder. We present a detailed discussion about the process of learning cross-lingual contextual representations, also about the performance in cross-lingual transfer learning and its implications.

Review:
This paper proposes a new approach to learning contextualised cross-lingual representations based on a three-step procedure called: transform-extract-reorder. The proposed model is tested on zero-shot cross-lingual transfer for one language pair: English-Chinese, where the gains are reported over the initial multilingual BERT model which does not rely on any bilingual signal. I like the main modeling idea, and the actual model is described quite nicely and in an intuitive way. However, I believe that much more work is needed in terms of proper experiments including: a) additional strong and insightful baselines; b) experiments with more language pairs; c) experiments across other transfer tasks. The paper also does not include nor compares to all relevant previous work. My main comments and remarks are as follows:
*Related work and Novelty* The topic of learning contextualised cross-lingual representations has become quite popular since the rise of ELMo and BERT. However, the authors cite only one paper that proposed such contextualised xling representations (the work of Schuster of al.). However, they never compare to that model, which is quite weird given the fact that the main goal of all xling representation models is to enable cross-lingual transfer in the first place. A comparison to the cited work of Schuster et al., plus additional comparisons to other (non-cited) models are required to clearly understand the main empirical benefits of the proposed three-step framework. Other relevant papers which are not cited nor compared against: Mulcaire et al. (NAACL 2019); Aldarmaki and Diab (NAACL 2019).
The authors should really stress the key novelty of their approach in comparison to the existing literature (which is mostly based on simpler projection-based methods).
- Note that there are more papers on the subject getting published soon such as Mulcaire et al. (CoNLL 2019) and Liu et al. (CoNLL 2019); while making comparisons to models introduced in those papers is not required, it would be nice to also briefly mention that latest work in the related work section once it gets published.
*Comparisons to XLM* Based on the limited set of results reported in the paper, it seems that the proposed model cannot match the performance of the XLM model (Lample and Conneau, NeurIPS 2019). The authors try to justify the usage of their method over XLM by emphasising the fact that their method produces language-independent xling representations, which XLM cannot do. However, there are multiple issues here: 1) it is not clear why exactly this requirement of language-independence is needed for a task such as zero-shot XNLI; 2) the paper does not demonstrate the usefulness of that language-independence property empirically at all. To me, it just seems like an unsuccessful attempt to make a conceptual distinction between the proposed model and XLM given the fact that XLM seems to significantly outperform the proposed model.
Page 8 ("Training cost and beyond"). The authors claim that 250k pairs of parallel sentences is enough to learn a near-optimal cross-lingual model and that the model saturates with more parallel sentences. I don't see it as a positive aspect of the model as claimed by the authors. This means that the model cannot optimally encode knowledge about variety of cross-lingual contexts during its training, i.e., it stops learning earlier than expected. Why does that happen? A plot showing transfer learning results of XLM versus the proposed model versus some static word embedding model that also exploits parallel data would be very useful. The authors also do not even speculate why their model saturates already with 250k sentence pairs. Given the limited set of results (only one task and only language pair) it also remains unknown how general this phenomenon is and whether the same point is hit with some other language pairs and with other tasks. This definitely requires further and more thorough investigation.
*Other Comparisons and Experiments*
As mentioned, the paper is very limited when it comes to proper thorough evaluation. Most experiments focus on intrinsic/internal evaluations of different steps of the proposed framework, or some probing tests, which can also be seen simply as diagnostic experiments. The actual downstream experiments are conducted on only one language pair and for only one (zero-shot) transfer task. This is definitely not sufficient to draw any generalisable claims, and it prevents us to dig deeper into other interesting aspects of the model. I would suggest the authors to maybe run the model across the same range of tasks as done in the XLM work of Lample and Conneau, and definitely for more languages (ideally diverse target languages).
Also, besides XLM and translation-based baselines reported by multilingual BERT's github page, there is actually no other baselines, which really makes it hard to put this work in context, and understand its usefulness. For instance, static cross-lingual word embeddings could be added to ESIM and used to evaluate on XNLI as well (see, e.g., the work of Glavas et al. (ACL 2019)). It would be interesting to report the benefits of replacing such static vectors with truly contextualised representations. Also, as mentioned before, comparisons to other models that learn contextualised cross-lingual representations are definitely something that should be included in the paper.
*Other Comments*
- Based on the result from Table 4, it seems that, in order to enable fully language-independent representations, one must sacrifice some of the monolingual performance, as the numbers drop from T over T+E to T+E+R variant. Is the same pattern visible for other languages monolingually? Why does this happen? 
- As mentioned before, the whole emphasis is of language-independence is somewhat oversold in the whole paper without providing sufficient empirical evidence that this is crucial for transfer performance.