Paper:
We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, i.e. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

Review:
This paper conducts a series of experiments on the multilingual BERT model of Devlin et al., aiming to inject stronger bilingual knowledge into the model for improved 'Aligned BERT'. The knowledge originating from parallel (Europarl) data improves the model significantly as shown on tasks such as contextual and non-contextual word retrieval as well as in zero-shot XNLI task. The paper continues the line of work on cross-lingual contextualised word embeddings, and it brings several minor contributions, but overall I do not see it as a very inspiring piece of work, and it leaves open several very important questions, in particular its relationship to prior work and some potentially stronger baselines than the ones reported in the paper, plus more experiments with more distant language pairs.
I am not exactly sure that the comparison between 'Aligned BERT' and the main baseline 'Aligned fastText + sentence' is completely fair. 'Aligned BERT' uses more than 2M Europarl sentences to learn the alignment, while the standard alignment methods for learning cross-lingual word embeddings (see e.g. Ruder et al.'s survey) typically rely only on 5k translation pairs or even less pairs. There is a huge difference in the strength of the bilingual signal between 2M parallel sentences and, say, 2k, word translation pairs. 
The main goal of the paper is to improve alignment of the starting multilingual BERT model, but I wonder why the authors have not compared to a more suitable XLM baseline of Lample and Conneau (NeurIPS 2019; the paper has been on arXiv since January 2019) - the XLM model uses exactly the same resources as 'Aligned BERT': parallel sentences from Europarl, while the main baseline here uses only seed dictionaries to learn the mapping. Regarding the baselines, it is also not clear to me why the authors have not compared to previous work of Schuster et al. (2019) and Aldarmaki and Diab (2019) at least in tasks where the models can be directly compared (XNLI or non-contextual word retrieval). Also, another non-contextual model which is worth trying is a joint model which relies on parallel sentences (similar to Ormazabal et al., ACL-19).
For the 'Aligned fastText + sentence' baseline, it would be interesting to report numbers with another (hybrid) baseline model that combines aligned fastText vectors with sentence encodings produced by multilingual BERT or some other multilingual sentence encoder (such as LASER, see Schwenk et al., 2019). Simply taking min, max, and avg vectors over all the sentence words might not be the best way to encode the sentence, and I would like to see more experiments here.
The paper makes some claims on novelty which 1) partially overlap with prior work, or 2) it does not cite related work while it leans on its findings. For instance on Page 4, the authors claim that their "(...) alignment method departs from prior work, in which each non-English language is rotated to match the English embedding space through individual learned matrices." However, there is at least one previous paper (Heyman et al., NAACL 2019) which did the same thing as the authors and showed that departing from learning projections only to English leads to more robust multilingual embeddings. Further, also on Page 4, the authors discuss that the assumption on learning good rotation matrices relies on the assumption of rough/approximate isomorphism without citing a body of related work that actually investigated this assumption such as the work of Sogaard et al. (ACL 2018). Also, the paper should do a better job in Section 2 and cover "word vector alignment" in more detail (e.g., a good starting point might be Ruder et al.'s survey paper on cross-lingual word embeddings).
The assumption of rough/approximate isomorphism is problematic also for non-contextual cross-lingual embeddings in settings with more distant language pairs. The authors mention that it may not hold for 'contextual pre-trained models given their increased complexity'. This is very imprecise writing taking place imho: 1) it is not clear why it should not hold in the case of contextual pre-trained models (at least for similar languages). Are there any properties of the contextual models that invalidate that assumption? It is also not exactly shown why contextual pre-trained models have increased complexity compared to e.g. fastText. How does one measure that 'model complexity' in objective terms? In fact, the paper would contribute immensely from more precise writing: e.g., on Page 3 contextual alignment of the model f is defined as accuracy in contextual word retrieval. This reads as defining a critical concept or a task as an evaluation measure (that measures the success of that task). In Introduction, the paper aims to "better understand BERT’s multilingualism", but I do not see how it contributes to our better understanding of BERT's multilingualism besides a pretty straightforward claim that it shows less multilingual potential when doing experiments with Greek and Bulgarian that use different scripts. Figure 2 and Figure 3 also do not bring anything new - the paper seems to just state known facts without proposing new solutions on how to e.g. learn better alignments for Greek or Bulgarian.
One important analysis aspect is missing from the paper: there are no experiments with more distant language pairs (the most distant language pair is English-Greek). I would like to see more experiments in this space. Another experiment which would contribute to the paper is the analysis of the importance of parallel corpora size. How much does the model lose in its performance by shrinking the parallel corpus? We cannot expect having 2M sentences for so many language pairs, and, even if we do have the data, the paper does not convince me that I should use 'Aligned BERT' instead of e.g. the XLM model of Lample and Conneau.
Minor remarks:
As a variant of the contextual word retrieval, have the authors tested if a correct target language sentence can be retrieved only looking at the context of the source language word? This would provide some insight on the importance of modeling context via BERT versus via simple context averaging.
Regarding the analysis between closed-class and open-class words performance, the difference in performance can be due to mere frequency: closed-class word types are very scarce, but their corpus frequency is quite high which also leads to learning better representations in the first place, as well as better alignments later on.