Original: M ULTILINGUAL A LIGNMENT OF C ONTEXTUAL W ORD R EPRESENTATIONS
Revised: M ULTILINGUAL A LIGNMENT OF C ONTEXTUAL W ORD R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual
Revised: We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual

Original: BERT.
Revised: BERT.

Original: In particular, after our proposed alignment procedure, BERT exhibits signiﬁcantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek.
Revised: In particular, after our proposed alignment procedure, BERT exhibits signiﬁcantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek.

Original: Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer.
Revised: Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer.

Original: Using this word retrieval task, we also analyze BERT and ﬁnd that it exhibits systematic deﬁciencies, i.e. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure.
Revised: Using this word retrieval task, we also analyze BERT and ﬁnd that it exhibits systematic deﬁciencies, i.e. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure.

Original: These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.
Revised: These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Embedding alignment was originally studied for word vectors with the goal of enabling cross-lingual transfer, where the embeddings for two languages are in alignment if word translations, e.g. cat and
Revised: Embedding alignment was originally studied for word vectors with the goal of enabling cross-lingual transfer, where the embeddings for two languages are in alignment if word translations, e.g. cat and

Original: Katze , have similar representations (Mikolov et al., 2013a; Smith et al., 2017).
Revised: Katze , have similar representations (Mikolov et al., 2013a; Smith et al., 2017).

Original: Recently, large pretrained models have largely subsumed word vectors based on their accuracy on downstream tasks, partly due to the fact that their word representations are context-dependent, allowing them to more richly capture the meaning of a word (Peters et al., 2018; Howard & Ruder, 2018;
Revised: Recently, large pretrained models have largely subsumed word vectors based on their accuracy on downstream tasks, partly due to the fact that their word representations are context-dependent, allowing them to more richly capture the meaning of a word (Peters et al., 2018; Howard & Ruder, 2018;

Original: Radford et al., 2018;
Revised: Radford et al., 2018;

Original: Devlin et al., 2018).
Revised: Devlin et al., 2018).

Original: Therefore, with the same goal of cross-lingual transfer but for these more complex models, we might consider contextual embedding alignment, where we observe whether word pairs within parallel sentences, e.g. cat in “The cat sits” and Katze in “Die Katze sitzt,” have similar representations.
Revised: Therefore, with the same goal of cross-lingual transfer but for these more complex models, we might consider contextual embedding alignment, where we observe whether word pairs within parallel sentences, e.g. cat in “The cat sits” and Katze in “Die Katze sitzt,” have similar representations.

Original: One model relevant to these questions is multilingual BERT, a version of BERT pre-trained onlanguages that achieves remarkable transfer on downstream tasks.
Revised: One model relevant to these questions is multilingual BERT, a version of BERT pre-trained onlanguages that achieves remarkable transfer on downstream tasks.

Original: For example, after the model is ﬁne-tuned on the English MultiNLI training set, it achieves 74.3% accuracy on the test set in Spanish, which is only 7.1% lower than the English accuracy (Devlin et al., 2018; Conneau et al., 2018b).
Revised: For example, after the model is ﬁne-tuned on the English MultiNLI training set, it achieves 74.3% accuracy on the test set in Spanish, which is only 7.1% lower than the English accuracy (Devlin et al., 2018; Conneau et al., 2018b).

Original: Furthermore, while the model transfers better to languages similar to English, it still achieves reasonable accuracies even on languages with different scripts.
Revised: Furthermore, while the model transfers better to languages similar to English, it still achieves reasonable accuracies even on languages with different scripts.

Original: However, given the way that multilingual BERT was pre-trained, it is unclear why we should expect such high zero-shot performance.
Revised: However, given the way that multilingual BERT was pre-trained, it is unclear why we should expect such high zero-shot performance.

Original: Compared to monolingual BERT which exhibits no zero-shot transfer, multilingual BERT differs only in that (1) during pre-training (i.e. masked word prediction), each batch contains sentences from all of the languages, and (2) it uses a single shared vocabulary, formed by WordPiece on the concatenated monolingual corpora (Devlin et al., 2019).
Revised: Compared to monolingual BERT which exhibits no zero-shot transfer, multilingual BERT differs only in that (1) during pre-training (i.e. masked word prediction), each batch contains sentences from all of the languages, and (2) it uses a single shared vocabulary, formed by WordPiece on the concatenated monolingual corpora (Devlin et al., 2019).

Original: Therefore, we might wonder: (1) How can we better understand BERT’s multilingualism?
Revised: Therefore, we might wonder: (1) How can we better understand BERT’s multilingualism?

Original: (2) Can we further improve BERT’s cross-lingual transfer?
Revised: (2) Can we further improve BERT’s cross-lingual transfer?

Original: In this paper, we show that contextual embedding alignment is a useful concept for addressing these questions.
Revised: In this paper, we show that contextual embedding alignment is a useful concept for addressing these questions.

Original: First, we propose a contextual version of word retrieval to evaluate the degree of alignment, where a model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must ﬁnd the correct word and sentence in the other.
Revised: First, we propose a contextual version of word retrieval to evaluate the degree of alignment, where a model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must ﬁnd the correct word and sentence in the other.

Original: Using this metric of alignment, we show that multilingual BERT achieves zero-shot transfer because its embeddings are partially aligned, as depicted in Figure 1, with the degree of alignment predicting the degree of downstream transfer.
Revised: Using this metric of alignment, we show that multilingual BERT achieves zero-shot transfer because its embeddings are partially aligned, as depicted in Figure 1, with the degree of alignment predicting the degree of downstream transfer.

Original: Next, using 250K sentences per language from the Europarl corpus as parallel data (Koehn, 2005), we propose a ﬁne-tuning-based alignment procedure and show that it signiﬁcantly improves BERT as a multilingual model.
Revised: Next, using 250K sentences per language from the Europarl corpus as parallel data (Koehn, 2005), we propose a ﬁne-tuning-based alignment procedure and show that it signiﬁcantly improves BERT as a multilingual model.

Original: Speciﬁcally, on zero-shot XNLI, where the model is trained on English
Revised: Speciﬁcally, on zero-shot XNLI, where the model is trained on English

Original: MultiNLI and tested on other languages (Conneau et al., 2018b), the aligned model improves accuracies by 2.78% on average over the base model, and it remarkably matches translate-train models for Bulgarian and Greek, which approximate the fully-supervised setting.
Revised: MultiNLI and tested on other languages (Conneau et al., 2018b), the aligned model improves accuracies by 2.78% on average over the base model, and it remarkably matches translate-train models for Bulgarian and Greek, which approximate the fully-supervised setting.

Original: To put our results in the context of past work, we also use word retrieval to compare our ﬁnetuning procedure to two alternatives: (1) fastText augmented with sentence and aligned using rotations (Bojanowski et al., 2017; R ¨ uckl ´ e et al., 2018; Artetxe et al., 2018), and (2) BERT aligned using rotations (Aldarmaki & Diab, 2019;
Revised: To put our results in the context of past work, we also use word retrieval to compare our ﬁnetuning procedure to two alternatives: (1) fastText augmented with sentence and aligned using rotations (Bojanowski et al., 2017; R ¨ uckl ´ e et al., 2018; Artetxe et al., 2018), and (2) BERT aligned using rotations (Aldarmaki & Diab, 2019;

Original: Schuster et al., 2019; Wang et al., 2019).
Revised: Schuster et al., 2019; Wang et al., 2019).

Original: We ﬁnd that when there are multiple occurences per word, ﬁne-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT.
Revised: We ﬁnd that when there are multiple occurences per word, ﬁne-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT.

Original: This result supports the intuition that contextual alignment is more difﬁcult than its non-contextual counterpart, given that a rotation, at least when applied naively, is no longer sufﬁcient to produce strong alignments.
Revised: This result supports the intuition that contextual alignment is more difﬁcult than its non-contextual counterpart, given that a rotation, at least when applied naively, is no longer sufﬁcient to produce strong alignments.

Original: In addition, when there is only one occurrence per word, ﬁne-tuned BERT matches the performance of fastText.
Revised: In addition, when there is only one occurrence per word, ﬁne-tuned BERT matches the performance of fastText.

Original: Given that context disambiguation is no longer necessary, this result suggests that our ﬁne-tuning procedure is able to align BERT at the type level to a degree that matches non-contextual approaches.
Revised: Given that context disambiguation is no longer necessary, this result suggests that our ﬁne-tuning procedure is able to align BERT at the type level to a degree that matches non-contextual approaches.

Original: Finally, we use the contextual word retrieval task to conduct ﬁner-grained analysis of multilingual
Revised: Finally, we use the contextual word retrieval task to conduct ﬁner-grained analysis of multilingual

Original: BERT, with the goal of better understanding its strengths and shortcomings.
Revised: BERT, with the goal of better understanding its strengths and shortcomings.

Original: Speciﬁcally, we ﬁnd that base BERT has trouble aligning open-class compared to closed-class parts-of-speech, as well as word pairs that have large differences in usage frequency, suggesting insight into the pre-training procedure that we explore in Section 5.
Revised: Speciﬁcally, we ﬁnd that base BERT has trouble aligning open-class compared to closed-class parts-of-speech, as well as word pairs that have large differences in usage frequency, suggesting insight into the pre-training procedure that we explore in Section 5.

Original: Together, these experiments support contextual alignment as an important task that provides useful insight into large multilingual pre-trained models.
Revised: Together, these experiments support contextual alignment as an important task that provides useful insight into large multilingual pre-trained models.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Word vector alignment.
Revised: Word vector alignment.

Original: There has been a long line of works that learn aligned word vectors from varying levels of supervision (Ruder et al., 2019).
Revised: There has been a long line of works that learn aligned word vectors from varying levels of supervision (Ruder et al., 2019).

Original: One popular family of methods most relevant to our work starts with word vectors learned independently for each language using a method like skipgram with negative sampling (Mikolov et al., 2013b), and it learns a mapping from source language vectors to target language vectors with a bilingual dictionary as supervision (Mikolov et al., 2013a;
Revised: One popular family of methods most relevant to our work starts with word vectors learned independently for each language using a method like skipgram with negative sampling (Mikolov et al., 2013b), and it learns a mapping from source language vectors to target language vectors with a bilingual dictionary as supervision (Mikolov et al., 2013a;

Original: Smith et al., 2017; Artetxe et al., 2017).
Revised: Smith et al., 2017; Artetxe et al., 2017).

Original: When the mapping is constrained to be a orthogonal linear transformation, the optimal mapping that minimizes distances between word pairs can be solved in closed form (Artetxe et al., 2016; Schonemann, 1966).
Revised: When the mapping is constrained to be a orthogonal linear transformation, the optimal mapping that minimizes distances between word pairs can be solved in closed form (Artetxe et al., 2016; Schonemann, 1966).

Original: Because alignment is evaluated using bilingual lexicon induction, these papers also propose ways to mitigate the hubness problem in nearest neighbors, e.g. by using alternate similarity functions like CSLS (Conneau et al., 2018a).
Revised: Because alignment is evaluated using bilingual lexicon induction, these papers also propose ways to mitigate the hubness problem in nearest neighbors, e.g. by using alternate similarity functions like CSLS (Conneau et al., 2018a).

Original: A recent set of works has also shown that the mapping can be learned with minimal to no supervision by starting with some minimal seed dictionary and alternating between learning the linear map and inducing the dictionary (Artetxe et al., 2018;
Revised: A recent set of works has also shown that the mapping can be learned with minimal to no supervision by starting with some minimal seed dictionary and alternating between learning the linear map and inducing the dictionary (Artetxe et al., 2018;

Original: Conneau et al., 2018a; Hoshen & Wolf, 2018;
Revised: Conneau et al., 2018a; Hoshen & Wolf, 2018;

Original: Xu et al., 2018; Chen & Cardie, 2018).
Revised: Xu et al., 2018; Chen & Cardie, 2018).

Original: Incorporating context into alignment.
Revised: Incorporating context into alignment.

Original: One key challenge in making alignment context aware is that the embeddings are now different across multiple occurrences of the same word.
Revised: One key challenge in making alignment context aware is that the embeddings are now different across multiple occurrences of the same word.

Original: Past papers have handled this issue by removing context and aligning the “average sense” of a word.
Revised: Past papers have handled this issue by removing context and aligning the “average sense” of a word.

Original: In one such study, Schuster et al.
Revised: In one such study, Schuster et al.

Original: (2019) learn a rotation to align contextual ELMo embeddings (Peters et al., 2018) with the goal of improving zero-shot multilingual dependency parsing, and they handle context by taking the average embedding for a word in all of its contexts.
Revised: (2019) learn a rotation to align contextual ELMo embeddings (Peters et al., 2018) with the goal of improving zero-shot multilingual dependency parsing, and they handle context by taking the average embedding for a word in all of its contexts.

Original: In another paper, Aldarmaki & Diab (2019) learn a rotation on sentence vectors, produced by taking the average word vector over the sentence, and they show that the resulting alignment also works well for word-level tasks.
Revised: In another paper, Aldarmaki & Diab (2019) learn a rotation on sentence vectors, produced by taking the average word vector over the sentence, and they show that the resulting alignment also works well for word-level tasks.

Original: In a contemporaneous work, Wang et al.
Revised: In a contemporaneous work, Wang et al.

Original: (2019) align not only the word but also the context by learning a linear transformation using word-aligned parallel data to align multilingual BERT, with the goal of improving zero-shot dependency parsing numbers.
Revised: (2019) align not only the word but also the context by learning a linear transformation using word-aligned parallel data to align multilingual BERT, with the goal of improving zero-shot dependency parsing numbers.

Original: In this paper, we similarly align not only the word but also the context, and we also depart from these past works by using more expressive alignment methods than rotation.
Revised: In this paper, we similarly align not only the word but also the context, and we also depart from these past works by using more expressive alignment methods than rotation.

Original: Incorporating parallel texts into pre-training.
Revised: Incorporating parallel texts into pre-training.

Original: Instead of performing alignment post-hoc, another line of works proposes contextual pre-training procedures that are more cross-lingually-aware.
Revised: Instead of performing alignment post-hoc, another line of works proposes contextual pre-training procedures that are more cross-lingually-aware.

Original: Wieting et al.
Revised: Wieting et al.

Original: (2019) pre-train sentence embeddings using parallel texts by maximizing similarity between sentence pairs while minimizing similarity with negative examples.
Revised: (2019) pre-train sentence embeddings using parallel texts by maximizing similarity between sentence pairs while minimizing similarity with negative examples.

Original: Lample & Conneau (2019) propose a cross-lingual pre-training objective that incorporates parallel data in addition to monolingual corpora, leading to improved downstream cross-lingual transfer.
Revised: Lample & Conneau (2019) propose a cross-lingual pre-training objective that incorporates parallel data in addition to monolingual corpora, leading to improved downstream cross-lingual transfer.

Original: In contrast, our method uses less parallel data and aligns existing pre-trained models rather than requiring pretraining from scratch.
Revised: In contrast, our method uses less parallel data and aligns existing pre-trained models rather than requiring pretraining from scratch.

Original: Analyzing multilingual BERT.
Revised: Analyzing multilingual BERT.

Original: Pires et al.
Revised: Pires et al.

Original: (2019) present a series of probing experiments to better understand multilingual BERT, and they ﬁnd that transfer is possible even between dissimilar languages, but that it works better on languages that are typologically similar.
Revised: (2019) present a series of probing experiments to better understand multilingual BERT, and they ﬁnd that transfer is possible even between dissimilar languages, but that it works better on languages that are typologically similar.

Original: They conclude that
Revised: They conclude that

Original: BERT is remarkably multilingual but falls short for certain language pairs.
Revised: BERT is remarkably multilingual but falls short for certain language pairs.

Original:  M ETHODS 3. M ULTILINGUAL P RE -T RAINING
Revised:  M ETHODS 3. M ULTILINGUAL P RE -T RAINING

Original: We ﬁrst brieﬂy describe multilingual BERT (Devlin et al., 2018).
Revised: We ﬁrst brieﬂy describe multilingual BERT (Devlin et al., 2018).

Original: Like monolingual BERT, multilingual BERT is pre-trained on sentences from Wikipedia to perform two tasks: masked word prediction, where it must predict words that are masked within a sentence, and next sentence prediction, where it must predict whether the second sentence follows the ﬁrst one.
Revised: Like monolingual BERT, multilingual BERT is pre-trained on sentences from Wikipedia to perform two tasks: masked word prediction, where it must predict words that are masked within a sentence, and next sentence prediction, where it must predict whether the second sentence follows the ﬁrst one.

Original: The model is trained on 104 languages, with each batch containing training sentences from each language, and it uses a shared vocabulary formed by WordPiece on the 104 Wikipedias concatenated (Wu et al., 2016).
Revised: The model is trained on 104 languages, with each batch containing training sentences from each language, and it uses a shared vocabulary formed by WordPiece on the 104 Wikipedias concatenated (Wu et al., 2016).

Original: 3. D EFINING AND E VALUATING C ONTEXTUAL A LIGNMENT
Revised: 3. D EFINING AND E VALUATING C ONTEXTUAL A LIGNMENT

Original: In the following sections, we describe how to deﬁne, evaluate, and improve contextual alignment.
Revised: In the following sections, we describe how to deﬁne, evaluate, and improve contextual alignment.

Original: Given two languages, a model is in contextual alignment if it has similar representations for word pairs within parallel sentences.
Revised: Given two languages, a model is in contextual alignment if it has similar representations for word pairs within parallel sentences.

Original: More precisely, suppose we have N parallel sentences
Revised: More precisely, suppose we have N parallel sentences

Original: C = { ( s 1 , t 1 ) , ..., ( s N , t N ) } , where ( s , t ) is a source-target sentence pair.
Revised: C = { ( s 1 , t 1 ) , ..., ( s N , t N ) } , where ( s , t ) is a source-target sentence pair.

Original: Also, let each sentence pair ( s , t ) have word pairs, denoted a ( s , t )
Revised: Also, let each sentence pair ( s , t ) have word pairs, denoted a ( s , t )

Original: = { ( i 1 , j 1 ) , ..., ( i m , j m ) } , containing position tuples ( i, j ) such that the words s i and t j are translations of each other.
Revised: = { ( i 1 , j 1 ) , ..., ( i m , j m ) } , containing position tuples ( i, j ) such that the words s i and t j are translations of each other.

Original: 1 We will use f to represent a pre-trained model such that f ( i, s ) is the contextual embedding for the i th word in s .
Revised: 1 We will use f to represent a pre-trained model such that f ( i, s ) is the contextual embedding for the i th word in s .

Original: As an example, we might have the following sentence pair:
Revised: As an example, we might have the following sentence pair:

Original: [Equation]
Revised: [Equation]

Original: Then, using the parallel corpus C , we can measure the contextual alignment of the model f using its accuracy in contextual word retrieval .
Revised: Then, using the parallel corpus C , we can measure the contextual alignment of the model f using its accuracy in contextual word retrieval .

Original: In this task, the model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must ﬁnd the correct word and sentence in the other.
Revised: In this task, the model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must ﬁnd the correct word and sentence in the other.

Original: Speciﬁcally, we can deﬁne a nearest neighbor retrieval function
Revised: Speciﬁcally, we can deﬁne a nearest neighbor retrieval function

Original: [Equation]
Revised: [Equation]

Original: where i and j denote the position within a sentence and sim is a similarity function.
Revised: where i and j denote the position within a sentence and sim is a similarity function.

Original: The accuracy is then given by the percentage of exact matches over the entire corpus, or
Revised: The accuracy is then given by the percentage of exact matches over the entire corpus, or

Original: [Equation]
Revised: [Equation]

Original: where I represents the indicator function.
Revised: where I represents the indicator function.

Original: We can perform the same procedure in the other direction, where we pull target words given source words, so we report the average between the two directions.
Revised: We can perform the same procedure in the other direction, where we pull target words given source words, so we report the average between the two directions.

Original: As our similarity function, we use CSLS, a modiﬁed version of cosine similarity that mitigates the hubness problem, with neighborhood size 10 (Conneau et al., 2018a).
Revised: As our similarity function, we use CSLS, a modiﬁed version of cosine similarity that mitigates the hubness problem, with neighborhood size 10 (Conneau et al., 2018a).

Original: One additional point is that this procedure can be made more or less contextual based on the corpus: a corpus with more occurrences for each word type requires better representations of context.
Revised: One additional point is that this procedure can be made more or less contextual based on the corpus: a corpus with more occurrences for each word type requires better representations of context.

Original: Therefore, we also test non-contextual word retrieval by removing all but the ﬁrst occurrence of each word type.
Revised: Therefore, we also test non-contextual word retrieval by removing all but the ﬁrst occurrence of each word type.

Original: Given parallel data, these word pairs can be procured in an unsupervised fashion using standard techniques developed by the machine translation community (Brown et al., 1993).
Revised: Given parallel data, these word pairs can be procured in an unsupervised fashion using standard techniques developed by the machine translation community (Brown et al., 1993).

Original: While these methods can be noisy, by running the algorithm in both the source-target and target-source directions and only keeping word pairs in their intersection, we can trade-off coverage for accuracy, producing a reasonably high-precision dataset (Och & Ney, 2003).
Revised: While these methods can be noisy, by running the algorithm in both the source-target and target-source directions and only keeping word pairs in their intersection, we can trade-off coverage for accuracy, producing a reasonably high-precision dataset (Och & Ney, 2003).

Original: 3. A LIGNING P RE -T RAINED C ONTEXTUAL E MBEDDINGS
Revised: 3. A LIGNING P RE -T RAINED C ONTEXTUAL E MBEDDINGS

Original: To improve the alignment of the model f with respect to the corpus C , we can encapsulate alignment in the loss function
Revised: To improve the alignment of the model f with respect to the corpus C , we can encapsulate alignment in the loss function

Original: [Equation]
Revised: [Equation]

Original: where we sum the similarities between word pairs.
Revised: where we sum the similarities between word pairs.

Original: Because the CSLS metric is not easily optimized, we instead use the squared error loss, or sim( f ( i, s ) , f ( j, t ))
Revised: Because the CSLS metric is not easily optimized, we instead use the squared error loss, or sim( f ( i, s ) , f ( j, t ))

Original: =
Revised: =

Original: −|| f ( i, s ) − f ( j, t )
Revised: −|| f ( i, s ) − f ( j, t )

Original: || 22 .
Revised: || 22 .

Original: However, note that this loss function does not account for the informativity of f ; for example, it is zero if f is constant.
Revised: However, note that this loss function does not account for the informativity of f ; for example, it is zero if f is constant.

Original: Therefore, at a high level, we would like to minimize L ( f ; C ) while maintaining some aspect of f that makes it useful, e.g. its high accuracy when ﬁne-tuned on downstream tasks.
Revised: Therefore, at a high level, we would like to minimize L ( f ; C ) while maintaining some aspect of f that makes it useful, e.g. its high accuracy when ﬁne-tuned on downstream tasks.

Original: Letting f 0 denote the initial pre-trained model before alignment, we achieve this goal by deﬁning a regularization term
Revised: Letting f 0 denote the initial pre-trained model before alignment, we achieve this goal by deﬁning a regularization term

Original: [Equation]
Revised: [Equation]

Original: which imposes a penalty if the target language embeddings stray from their initialization
Revised: which imposes a penalty if the target language embeddings stray from their initialization

Original: Then, we sample minibatches B ⊂ C and take gradient steps of the function L ( f ; B ) + λR ( f ; B ) directly on the weights of f , which moves the source embeddings toward the target embeddings while preventing the latter from drifting too far.
Revised: Then, we sample minibatches B ⊂ C and take gradient steps of the function L ( f ; B ) + λR ( f ; B ) directly on the weights of f , which moves the source embeddings toward the target embeddings while preventing the latter from drifting too far.

Original: In our experiments, we set λ = 1 .
Revised: In our experiments, we set λ = 1 .

Original: In the multilingual case, suppose we have k parallel corpora C 1 , ..., C k , where each corpus has a different source language with the target language as English.
Revised: In the multilingual case, suppose we have k parallel corpora C 1 , ..., C k , where each corpus has a different source language with the target language as English.

Original: Then, we sample equal-sized batches
Revised: Then, we sample equal-sized batches

Original: B i ⊂ C i from each corpus and take gradient steps on (cid:80)
Revised: B i ⊂ C i from each corpus and take gradient steps on (cid:80)

Original: ki =1 L ( f ; B i ) + λR ( f ; B i ) , which moves all of the non-English embeddings toward English.
Revised: ki =1 L ( f ; B i ) + λR ( f ; B i ) , which moves all of the non-English embeddings toward English.

Original: Note that this alignment method departs from prior work, in which each non-English language is rotated to match the English embedding space through individual learned matrices.
Revised: Note that this alignment method departs from prior work, in which each non-English language is rotated to match the English embedding space through individual learned matrices.

Original: Speciﬁcally, the most widely used post-hoc alignment method learns a rotation W applied to the source vectors to minimize the distance between parallel word pairs, or
Revised: Speciﬁcally, the most widely used post-hoc alignment method learns a rotation W applied to the source vectors to minimize the distance between parallel word pairs, or

Original: [Equation]
Revised: [Equation]

Original: This problem is known as the Procrustes problem and can be solved in closed form (Schonemann, 1966).
Revised: This problem is known as the Procrustes problem and can be solved in closed form (Schonemann, 1966).

Original: This approach has the nice property that the vectors are only rotated, preserving distances and therefore the semantic information captured by the embeddings (Artetxe et al., 2016).
Revised: This approach has the nice property that the vectors are only rotated, preserving distances and therefore the semantic information captured by the embeddings (Artetxe et al., 2016).

Original: However, rotation requires the strong assumption that the embedding spaces are roughly isometric (Søgaard et al., 2018), an assumption that may not hold for contextual pre-trained models because they represent more aspects of a word than just its type, i.e. context and syntax, which are less likely to be isomorphic between languages.
Revised: However, rotation requires the strong assumption that the embedding spaces are roughly isometric (Søgaard et al., 2018), an assumption that may not hold for contextual pre-trained models because they represent more aspects of a word than just its type, i.e. context and syntax, which are less likely to be isomorphic between languages.

Original: Given that past work has also found independent alignment per language pair to be inferior to joint training (Heyman et al., 2019), another advantage of our method is that the alignment for all languages is done simultaneously.
Revised: Given that past work has also found independent alignment per language pair to be inferior to joint training (Heyman et al., 2019), another advantage of our method is that the alignment for all languages is done simultaneously.

Original: As our dataset, we use the Europarl corpora for English paired with Bulgarian, German, Greek,
Revised: As our dataset, we use the Europarl corpora for English paired with Bulgarian, German, Greek,

Original: Spanish, and French, the languages represented in both Europarl and XNLI (Koehn, 2005).
Revised: Spanish, and French, the languages represented in both Europarl and XNLI (Koehn, 2005).

Original: After tokenization (Koehn et al., 2007), we produce word pairs using fastAlign and keep the one-to-one pairs in the intersection (Dyer et al., 2013).
Revised: After tokenization (Koehn et al., 2007), we produce word pairs using fastAlign and keep the one-to-one pairs in the intersection (Dyer et al., 2013).

Original: We use the most recent 1024 sentences as the test set, the previous 1024 sentences as the development set, and the following 250K sentences as the training set.
Revised: We use the most recent 1024 sentences as the test set, the previous 1024 sentences as the development set, and the following 250K sentences as the training set.

Original: Furthermore, we modify the test set accuracy calculation to only include word pairs not seen in the training set.
Revised: Furthermore, we modify the test set accuracy calculation to only include word pairs not seen in the training set.

Original: We also remove any exact matches, e.g. punctuation and numbers, because BERT is already aligned for these pairs due to its shared vocabulary.
Revised: We also remove any exact matches, e.g. punctuation and numbers, because BERT is already aligned for these pairs due to its shared vocabulary.

Original: Given that parallel data may be limited for low-resource language pairs, we also report numbers for 10K and 50K parallel sentences.
Revised: Given that parallel data may be limited for low-resource language pairs, we also report numbers for 10K and 50K parallel sentences.

Original: 3. S ENTENCE -A UGMENTED N ON -C ONTEXTUAL B ASELINE
Revised: 3. S ENTENCE -A UGMENTED N ON -C ONTEXTUAL B ASELINE

Original: Given that there has been a long line of work on word vector alignment (Artetxe et al., 2018; Conneau et al., 2018a; Smith et al., 2017, inter alia ), we also compare BERT to a sentence-augmented fastText baseline (Bojanowski et al., 2017).
Revised: Given that there has been a long line of work on word vector alignment (Artetxe et al., 2018; Conneau et al., 2018a; Smith et al., 2017, inter alia ), we also compare BERT to a sentence-augmented fastText baseline (Bojanowski et al., 2017).

Original: Following Artetxe et al.
Revised: Following Artetxe et al.

Original: (2018), we ﬁrst normalize, then mean-center, then normalize the word vectors, and we then learn a rotation with the same parallel data as in the contextual case, as described in Equation 1.
Revised: (2018), we ﬁrst normalize, then mean-center, then normalize the word vectors, and we then learn a rotation with the same parallel data as in the contextual case, as described in Equation 1.

Original: We also strengthen this baseline by including sentence information: speciﬁcally, during word retrieval, we concatenate each word vector with a vector representing its sentence.
Revised: We also strengthen this baseline by including sentence information: speciﬁcally, during word retrieval, we concatenate each word vector with a vector representing its sentence.

Original: Following R ¨ uckl ´ e et al.
Revised: Following R ¨ uckl ´ e et al.

Original: (2018), we compute the sentence vector by concatenating the average, maximum, and minimum vector over all of the words in the sentence, a method that was shown to be state-of-the-art for a suite of cross-lingual tasks.
Revised: (2018), we compute the sentence vector by concatenating the average, maximum, and minimum vector over all of the words in the sentence, a method that was shown to be state-of-the-art for a suite of cross-lingual tasks.

Original: We also experimented with other methods, such as ﬁrst retrieving the sentence and then the word, but found this method resulted in the highest accuracy.
Revised: We also experimented with other methods, such as ﬁrst retrieving the sentence and then the word, but found this method resulted in the highest accuracy.

Original: As a result, the fastText vectors are 1200 -dimensional, while the BERT vectors are 768 -dimensional.
Revised: As a result, the fastText vectors are 1200 -dimensional, while the BERT vectors are 768 -dimensional.

Original: 3. T ESTING Z ERO -S HOT T RANSFER
Revised: 3. T ESTING Z ERO -S HOT T RANSFER

Original: The next step is to determine whether better alignment improves cross-lingual transfer.
Revised: The next step is to determine whether better alignment improves cross-lingual transfer.

Original: As our downstream task, we use the XNLI dataset, where the English MultiNLI development and test sets are human-translated into multiple languages (Conneau et al., 2018b; Williams et al., 2018).
Revised: As our downstream task, we use the XNLI dataset, where the English MultiNLI development and test sets are human-translated into multiple languages (Conneau et al., 2018b; Williams et al., 2018).

Original: Given a pair of sentences, the task is to predict whether the ﬁrst sentence implies the second, where there are three labels: entailment, neutral, or contradiction.
Revised: Given a pair of sentences, the task is to predict whether the ﬁrst sentence implies the second, where there are three labels: entailment, neutral, or contradiction.

Original: Starting from either the base or aligned multilingual BERT, we train on English and evaluate on Bulgarian, German, Greek, Spanish, and French, the XNLI languages represented in Europarl.
Revised: Starting from either the base or aligned multilingual BERT, we train on English and evaluate on Bulgarian, German, Greek, Spanish, and French, the XNLI languages represented in Europarl.

Original: As our architecture, following Devlin et al.
Revised: As our architecture, following Devlin et al.

Original: (2018), we apply a linear layer followed by softmax on the [CLS] embedding of the sentence pair, producing scores for each of the three labels.
Revised: (2018), we apply a linear layer followed by softmax on the [CLS] embedding of the sentence pair, producing scores for each of the three labels.

Original: The model is trained using cross-entropy loss and selected based on its development set accuracy averaged across all of the languages.
Revised: The model is trained using cross-entropy loss and selected based on its development set accuracy averaged across all of the languages.

Original: As a fully-supervised ceiling, we also compare to models trained and tested on the same language, where for the non-English training data, we use the machine translations of the English MultiNLI training data provided by Conneau et al.
Revised: As a fully-supervised ceiling, we also compare to models trained and tested on the same language, where for the non-English training data, we use the machine translations of the English MultiNLI training data provided by Conneau et al.

Original: (2018b).
Revised: (2018b).

Original: While the quality of the training data is affected by the quality of the MT system, this comparison nevertheless serves as a good approximation for the fully-supervised setting.
Revised: While the quality of the training data is affected by the quality of the MT system, this comparison nevertheless serves as a good approximation for the fully-supervised setting.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  R ESULTS 4. Z ERO - SHOT XNLI T RANSFER
Revised:  R ESULTS 4. Z ERO - SHOT XNLI T RANSFER

Original: First, we test whether alignment improves multilingual BERT by applying the models to zero-shot
Revised: First, we test whether alignment improves multilingual BERT by applying the models to zero-shot

Original: XNLI, as displayed in Table 1.
Revised: XNLI, as displayed in Table 1.

Original: We see that our alignment procedure greatly improves accuracies, with all languages seeing a gain of at least 1% .
Revised: We see that our alignment procedure greatly improves accuracies, with all languages seeing a gain of at least 1% .

Original: In particular, the Bulgarian and Greek zero-shot numbers are boosted by almost 5% each and match the translate-train numbers, suggesting that the alignment procedure is especially effective for languages that are initially difﬁcult for BERT.
Revised: In particular, the Bulgarian and Greek zero-shot numbers are boosted by almost 5% each and match the translate-train numbers, suggesting that the alignment procedure is especially effective for languages that are initially difﬁcult for BERT.

Revised: We also run alignment for more distant language pairs (Chinese, Arabic, Urdu) and ﬁnd similar results, which we report in the appendix.

Original: Comparing to rotation-based methods (Aldarmaki & Diab, 2019; Wang et al., 2019), we ﬁnd that a rotation produces small gains for some languages, namely Bulgarian, German, and Spanish, but is sub-optimal overall, providing evidence that the increased expressivity of our proposed procedure is beneﬁcial for contextual alignment.
Revised: Comparing to rotation-based methods (Aldarmaki & Diab, 2019; Wang et al., 2019), we ﬁnd that a rotation produces small gains for some languages, namely Bulgarian, German, and Spanish, but is sub-optimal overall, providing evidence that the increased expressivity of our proposed procedure is beneﬁcial for contextual alignment.

Original: We explore this comparison more in Section 5.1.
Revised: We explore this comparison more in Section 5.1.

Original: 4. A LIGNMENT WITH L ESS D ATA
Revised: 4. A LIGNMENT WITH L ESS D ATA

Original: Given that our goal is zero-shot transfer, we cannot expect to always have large amounts of parallel data.
Revised: Given that our goal is zero-shot transfer, we cannot expect to always have large amounts of parallel data.

Original: Therefore, we also characterize the performance of our alignment method with varying amounts of data, as displayed in Table 2.
Revised: Therefore, we also characterize the performance of our alignment method with varying amounts of data, as displayed in Table 2.

Original: We ﬁnd that it improves transfer with as little as 10K sentences per language, making it a promising approach for low-resource languages.
Revised: We ﬁnd that it improves transfer with as little as 10K sentences per language, making it a promising approach for low-resource languages.

Original: [Table]
Revised: [Table]

Original:  A NALYSIS 5. W ORD R ETRIEVAL
Revised:  A NALYSIS 5. W ORD R ETRIEVAL

Original: In the following sections
Revised: In the following sections

Original: [Table]
Revised: [Table]

Original: we present word retrieval results to both compare our method to past work and better understand the strengths and weaknesses of multilingual BERT.
Revised: we present word retrieval results to both compare our method to past work and better understand the strengths and weaknesses of multilingual BERT.

Original: Table 3 displays the word retrieval accuracies for the aligned sentence-augmented fastText baseline and BERT pre- and postalignment.
Revised: Table 3 displays the word retrieval accuracies for the aligned sentence-augmented fastText baseline and BERT pre- and postalignment.

Original: First, we ﬁnd that in contextual retrieval, ﬁne-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT.
Revised: First, we ﬁnd that in contextual retrieval, ﬁne-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT.

Original: This result supports the intuition that aligning large pre-trained models is more difﬁcult than aligning word vectors, given that a rotation, at least when applied naively, produces sub-par alignments.
Revised: This result supports the intuition that aligning large pre-trained models is more difﬁcult than aligning word vectors, given that a rotation, at least when applied naively, produces sub-par alignments.

Original: In addition, ﬁne-tuned BERT matches the performance of fastText in non-contextual retrieval, suggesting that our alignment procedure overcomes these challenges and achieves type-level alignment that matches non-contextual approaches.
Revised: In addition, ﬁne-tuned BERT matches the performance of fastText in non-contextual retrieval, suggesting that our alignment procedure overcomes these challenges and achieves type-level alignment that matches non-contextual approaches.

Original: In the appendix, we also provide examples of aligned BERT disambiguating between different meanings of a word, giving qualitative evidence of the beneﬁt of context alignment.
Revised: In the appendix, we also provide examples of aligned BERT disambiguating between different meanings of a word, giving qualitative evidence of the beneﬁt of context alignment.

Original: We also ﬁnd that before alignment, BERT’s performance varies greatly between languages, while after alignment it is consistently effective.
Revised: We also ﬁnd that before alignment, BERT’s performance varies greatly between languages, while after alignment it is consistently effective.

Original: In particular, Bulgarian and Greek initially have very low accuracies.
Revised: In particular, Bulgarian and Greek initially have very low accuracies.

Original: This phenomenon is also reﬂected in the XNLI numbers (Table 1), where Bulgarian and Greek receive the largest boosts from alignment.
Revised: This phenomenon is also reﬂected in the XNLI numbers (Table 1), where Bulgarian and Greek receive the largest boosts from alignment.

Original: Examining the connection between alignment and zero-shot more closely, we ﬁnd that the word retrieval accuracies are highly correlated with downstream zero-shot performance (Figure 2), supporting our evaluation measure as predictive of cross-lingual transfer.
Revised: Examining the connection between alignment and zero-shot more closely, we ﬁnd that the word retrieval accuracies are highly correlated with downstream zero-shot performance (Figure 2), supporting our evaluation measure as predictive of cross-lingual transfer.

Original: The language discrepancies are also consistent with a hypothesis by Pires et al.
Revised: The language discrepancies are also consistent with a hypothesis by Pires et al.

Original: (2019) to explain
Revised: (2019) to explain

Original: BERT’s multilingualism.
Revised: BERT’s multilingualism.

Original: They posit that due to the shared vocabulary, shared words between languages, e.g. numbers and names, are forced to have the same representation.
Revised: They posit that due to the shared vocabulary, shared words between languages, e.g. numbers and names, are forced to have the same representation.

Original: Then, due to the masked word prediction task, other words that co-occur with these shared words also receive similar representations.
Revised: Then, due to the masked word prediction task, other words that co-occur with these shared words also receive similar representations.

Original: If this hypothesis is true, then languages with higher lexical overlap with English are likely to experience higher transfer.
Revised: If this hypothesis is true, then languages with higher lexical overlap with English are likely to experience higher transfer.

Original: As an extreme form of this phenomenon, Bulgarian and Greek have completely different scripts and should experience worse transfer than the common-script languages, an intuition that is conﬁrmed by the word retrieval and XNLI accuracies.
Revised: As an extreme form of this phenomenon, Bulgarian and Greek have completely different scripts and should experience worse transfer than the common-script languages, an intuition that is conﬁrmed by the word retrieval and XNLI accuracies.

Original: The fact that all languages are equally aligned with English post-alignment suggests that the pre-training procedure is needlessly suboptimal for these languages.
Revised: The fact that all languages are equally aligned with English post-alignment suggests that the pre-training procedure is needlessly suboptimal for these languages.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: and target.
Revised: and target.

Original: The accuracy of base BERT plummets for larger differences, suggesting that
Revised: The accuracy of base BERT plummets for larger differences, suggesting that

Original: its alignment depends on word pairs having similar usage statistics
Revised: its alignment depends on word pairs having similar usage statistics

Original: 5. W ORD R ETRIEVAL P ART - OF -S PEECH A NALYSIS
Revised: 5. W ORD R ETRIEVAL P ART - OF -S PEECH A NALYSIS

Original: Next, to gain insight into the multilingual pre-training procedure, we analyze the accuracy broken down by part-of-speech using the Universal Part-of-Speech Tagset (Petrov et al., 2012), annotated using polyglot (Al-Rfou et al., 2013) for Bulgarian and spaCy (Honnibal & Montani, 2017) for the other languages, as displayed in Table 4.
Revised: Next, to gain insight into the multilingual pre-training procedure, we analyze the accuracy broken down by part-of-speech using the Universal Part-of-Speech Tagset (Petrov et al., 2012), annotated using polyglot (Al-Rfou et al., 2013) for Bulgarian and spaCy (Honnibal & Montani, 2017) for the other languages, as displayed in Table 4.

Original: Unsurprisingly, multilingual BERT has high alignment out-of-the-box for groups with high lexical overlap, e.g. numerals, punctuation, and proper nouns, due to its shared vocabulary.
Revised: Unsurprisingly, multilingual BERT has high alignment out-of-the-box for groups with high lexical overlap, e.g. numerals, punctuation, and proper nouns, due to its shared vocabulary.

Original: We further divide the remaining tags into closed-class and open-class, where closed-class parts-of-speech correspond to ﬁxed sets of words serving grammatical functions (e.g. determiner, preposition, conjunction, pronoun, and auxiliary), while open-class parts-of-speech correspond to lexical words (e.g. noun, adverb, adjective, verb).
Revised: We further divide the remaining tags into closed-class and open-class, where closed-class parts-of-speech correspond to ﬁxed sets of words serving grammatical functions (e.g. determiner, preposition, conjunction, pronoun, and auxiliary), while open-class parts-of-speech correspond to lexical words (e.g. noun, adverb, adjective, verb).

Original: Interestingly, we see that base BERT has consistently lower accuracy for closed-class versus open-class categories ( 0 .
Revised: Interestingly, we see that base BERT has consistently lower accuracy for closed-class versus open-class categories ( 0 .

Original: 70 vs 0 . 54 ), but that this discrepancy disappears after alignment ( 0 . 88 vs 0 . 89 ).
Revised: 70 vs 0 . 54 ), but that this discrepancy disappears after alignment ( 0 . 88 vs 0 . 89 ).

Original: 5. U SAGE H YPOTHESIS FOR A LIGNMENT
Revised: 5. U SAGE H YPOTHESIS FOR A LIGNMENT

Original: From this closed-class vs open-class difference, we hypothesize that BERT’s alignment of a particular word pair is inﬂuenced by the similarity of their usage statistics.
Revised: From this closed-class vs open-class difference, we hypothesize that BERT’s alignment of a particular word pair is inﬂuenced by the similarity of their usage statistics.

Original: Speciﬁcally, given that
Revised: Speciﬁcally, given that

Original: BERT is trained through masked word prediction, its embeddings are in large part determined by the co-occurrences between words.
Revised: BERT is trained through masked word prediction, its embeddings are in large part determined by the co-occurrences between words.

Original: Therefore, two words that are used in similar contexts should be better aligned.
Revised: Therefore, two words that are used in similar contexts should be better aligned.

Original: This hypothesis provides an explanation of the closed-class vs open-class difference: closed-class words are typically grammatical, so they are used in similar ways across typologically similar languages.
Revised: This hypothesis provides an explanation of the closed-class vs open-class difference: closed-class words are typically grammatical, so they are used in similar ways across typologically similar languages.

Original: Furthermore, these words cannot be substituted for one another due to their grammatical function.
Revised: Furthermore, these words cannot be substituted for one another due to their grammatical function.

Original: Therefore, their usage statistics are a strong signature that can be used for alignment.
Revised: Therefore, their usage statistics are a strong signature that can be used for alignment.

Original: On the other hand, open-class words can be substituted for one another: for example, in most sentences, the noun tokens could be replaced by a wide range of semantically dissimilar nouns with the sentence remaining syntactically well-formed.
Revised: On the other hand, open-class words can be substituted for one another: for example, in most sentences, the noun tokens could be replaced by a wide range of semantically dissimilar nouns with the sentence remaining syntactically well-formed.

Original: By this effect, many nouns have similar co-occurrences, making them difﬁcult to align through masked word prediction alone.
Revised: By this effect, many nouns have similar co-occurrences, making them difﬁcult to align through masked word prediction alone.

Original: To further test this hypothesis, we plot the word retrieval accuracy versus the difference between the frequency rank of the target and source word, where this difference measures discrepancies in usage, as depicted in Figure 3.
Revised: To further test this hypothesis, we plot the word retrieval accuracy versus the difference between the frequency rank of the target and source word, where this difference measures discrepancies in usage, as depicted in Figure 3.

Original: We see that accuracy drops off signiﬁcantly as the source-target difference increases, supporting our hypothesis.
Revised: We see that accuracy drops off signiﬁcantly as the source-target difference increases, supporting our hypothesis.

Original: Furthermore, this shortcoming is remedied by alignment, revealing another systematic deﬁciency of multilingual pre-training.
Revised: Furthermore, this shortcoming is remedied by alignment, revealing another systematic deﬁciency of multilingual pre-training.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Given that the degree of alignment is causally predictive of downstream cross-lingual transfer, contextual alignment proves to be a useful concept for understanding and improving multilingual pretrained models.
Revised: Given that the degree of alignment is causally predictive of downstream cross-lingual transfer, contextual alignment proves to be a useful concept for understanding and improving multilingual pretrained models.

Original: Given small amounts of parallel data, our alignment procedure improves multilingual BERT and corrects many of its systematic deﬁciencies.
Revised: Given small amounts of parallel data, our alignment procedure improves multilingual BERT and corrects many of its systematic deﬁciencies.

Original: Contextual word retrieval also provides useful new insights into the pre-training procedure, opening up new avenues for analysis
Revised: Contextual word retrieval also provides useful new insights into the pre-training procedure, opening up new avenues for analysis

Original: A A PPENDIX A. O PTIMIZATION H YPERPARAMETERS
Revised: A A PPENDIX A. O PTIMIZATION H YPERPARAMETERS

Original: For both alignment and XNLI optimization, we use a learning rate of 5 × 10 − 5 with Adam hyperparameters β
Revised: For both alignment and XNLI optimization, we use a learning rate of 5 × 10 − 5 with Adam hyperparameters β

Original: = (0 . 9 , 0 . 98) , (cid:15) = 10 − 9 and linear learning rate warmup for the ﬁrst 10% of the training data.
Revised: = (0 . 9 , 0 . 98) , (cid:15) = 10 − 9 and linear learning rate warmup for the ﬁrst 10% of the training data.

Original: For alignment, the model is trained for one epoch, with each batch containing 2 sentence pairs per language.
Revised: For alignment, the model is trained for one epoch, with each batch containing 2 sentence pairs per language.

Original: For XNLI, each model is trained for 3 epochs with 32 examples per batch, and 10% dropout is applied to the BERT embeddings.
Revised: For XNLI, each model is trained for 3 epochs with 32 examples per batch, and 10% dropout is applied to the BERT embeddings.

Original: A. M ORE E XAMPLES OF C ONTEXT -A WARE R ETRIEVAL
Revised: A. A LIGNMENT OF C HINESE , A RABIC , AND U RDU

Revised: In Table 5, we report numbers for additional languages, where we align a single BERT model for all eight languages and then ﬁne-tune on XNLI.

Revised: We use 20K sentences per language, where we use the

Revised: MultiUN corpus for Arabic and Chinese (Eisele & Chen, 2010), the Tanzil corpus for Urdu (Tiedemann, 2012), and the Europarl corpus for the other ﬁve languages (Koehn, 2005).

Revised: This result conﬁrms that the alignment method works for a variety of languages and corpora.

Revised: Furthermore, the

Revised: Tanzil corpus consists of sentences from the Quran, suggesting that the method works even when the parallel corpus and downstream task contain sentences from entirely different domains.

Revised: A. M ORE E XAMPLES OF C ONTEXT -A WARE R ETRIEVAL

Original: In this section, we qualitatively show that aligned BERT is able to disambiguate between different occurences of a word.
Revised: In this section, we qualitatively show that aligned BERT is able to disambiguate between different occurences of a word.

Original: First, we ﬁnd two meanings of the word “like” occurring in the EnglishGerman Europarl test set.
Revised: First, we ﬁnd two meanings of the word “like” occurring in the EnglishGerman Europarl test set.

Original: Note also that in the second and third example, the two senses of “like” occur in the same sentence.
Revised: Note also that in the second and third example, the two senses of “like” occur in the same sentence.

Original: • This empire did not look for colonies far from home or overseas, like most Western European States, but close by.
Revised: • This empire did not look for colonies far from home or overseas, like most Western European States, but close by.

Original: Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee wie die meisten westeurop¨aischen
Revised: Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee wie die meisten westeurop¨aischen

Original: Staaten , sondern in der unmittelbaren Umgebung
Revised: Staaten, sondern in der unmittelbaren Umgebung.

Original: • Like other speakers, I would like to support the call for the arms embargo to remain.
Revised: • Like other speakers, I would like to support the call for the arms embargo to remain.

Original: Wie andere Sprecher, so m¨ochte auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen.
Revised: Wie andere Sprecher, so m¨ochte auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen.

Original: • Like other speakers, I would like to support the call for the arms embargo to remain.
Revised: • Like other speakers, I would like to support the call for the arms embargo to remain.

Original: Wie andere Sprecher, so m¨ochte auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen.
Revised: Wie andere Sprecher, so m¨ochte auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen.

Original: • I would also like , although they are absent, to mention the Commission and the Council.
Revised: • I would also like , although they are absent, to mention the Commission and the Council.

Original: Ich m¨ochte mir sogar erlauben, die Kommission und den Rat zu nennen, auch wenn sie nicht anwesend sind
Revised: Ich m¨ochte mir sogar erlauben die Kommission und den Rat zu nennen, auch wenn sie nicht anwesend sind

Original: Multiple meanings of “order”:
Revised: Multiple meanings of “order”:

Original: • Moreover, the national political elite had to make a detour in Ambon in order to reach the civil governor’s residence by warship.
Revised: • Moreover, the national political elite had to make a detour in Ambon in order to reach the civil governor’s residence by warship.

Original: In Ambon mußte die politische Spitze des Landes auch noch einen
Revised: In Ambon mußte die politische Spitze des Landes auch noch einen

Original: Umweg machen um mit einem Kriegsschiff die Residenz des Provinzgouverneurs zu erreichen.
Revised: Umweg machen, um mit einem Kriegsschiff die Residenz des Provinzgouverneurs zu erreichen.

Original: • Although the European Union has an interest in being surrounded by large, stable regions, the tools it has available in order to achieve this are still very limited.
Revised: • Although the European Union has an interest in being surrounded by large, stable regions, the tools it has available in order to achieve this are still very limited.

Original: Der Europ¨aischen Union ist zwar an großen stabilen Regionen in ihrer Umgebung gelegen, aber sie verfgt nach wie vor nur ber recht begrenzte Instrumente, um das zu erreichen.
Revised: Der Europ¨aischen Union ist zwar an großen stabilen Regionen in ihrer Umgebung gelegen, aber sie verfgt nach wie vor nur ber recht begrenzte Instrumente, um das zu erreichen.

Original: • We could reasonably expect the new Indonesian government to take action in three fundamental areas: restoring public order , prosecuting and punishing those who have blood on their hands and entering into a political dialogue with the opposition.
Revised: • We could reasonably expect the new Indonesian government to take action in three fundamental areas: restoring public order , prosecuting and punishing those who have blood on their hands and entering into a political dialogue with the opposition.

Original: Von der neuen indonesischen Regierung darf man mit Fug und Recht drei elementare Maßnahmen erwarten: die Wiederherstellung der ¨ offentlichen Ordnung , die Verfolgung und Bestrafung derjenigen, an deren H ¨ anden Blut klebt, und die Aufnahme des politischen Dialogs mit den Gegnern.
Revised: Von der neuen indonesischen Regierung darf man mit Fug und Recht drei elementare Maßnahmen erwarten: die Wiederherstellung der ¨offentlichen Ordnung , die Verfolgung und Bestrafung derjenigen, an deren H¨anden Blut klebt, und die Aufnahme des politischen Dialogs mit den Gegnern.

Original: • Firstly, I might mention the fact that the army needs to be reformed, secondly that a stable system of law and order needs to be introduced.
Revised: • Firstly, I might mention the fact that the army needs to be reformed, secondly that a stable system of law and order needs to be introduced.

Original: Ich nenne hier an erster Stelle die notwendige Reform der Armee, ferner die Einfhrung eines stabilen Systems rechtsstaatlicher Ordnung .
Revised: Ich nenne hier an erster Stelle die notwendige Reform der Armee, ferner die Einfhrung eines stabilen Systems rechtsstaatlicher Ordnung .

Original: Multiple meanings of “support”:
Revised: Multiple meanings of “support”:

Original: • Financial support is needed to enable poor countries to take part in these court activities.
Revised: • Financial support is needed to enable poor countries to take part in these court activities.

Original: Arme L¨ander m¨ussen ﬁnanziell unterst¨utzt werden, damit auch sie sich an der Arbeit des Gerichtshofs beteiligen k¨onnen.
Revised: Arme L¨ander m¨ussen ﬁnanziell unterst¨utzt werden, damit auch sie sich an der Arbeit des Gerichtshofs beteiligen k¨onnen.

Original: • We must help them and ensure that a proper action plan is implemented to support their work.
Revised: • We must help them and ensure that a proper action plan is implemented to support their work.

Original: Es gilt einen wirklichen
Revised: Es gilt einen wirklichen

Original: Aktionsplan auf den Weg zu bringen, um die Arbeit dieser Organisationen zu unterst¨utzen .
Revised: Aktionsplan auf den Weg zu bringen, um die Arbeit dieser Organisationen zu unterst¨utzen .

Original: So I hope that you will all support this resolution condemning the abominable conditions of prisoners and civilians in Djibouti.
Revised: So I hope that you will all support this resolution condemning the abominable conditions of prisoners and civilians in Djibouti.

Original: Ich hoffe daher, daß Sie alle diese Entschließung bef¨urworten , die die entsetzlichen
Revised: Ich hoffe daher, daß Sie alle diese Entschließung bef¨urworten , die die entsetzlichen

Original: Bedingungen von Inhaftierten und Zivilpersonen in Dschibuti verurteilt.
Revised: Bedingungen von Inhaftierten und Zivilpersonen in Dschibuti verurteilt

Original: • It would be difﬁcult to support a subsidy scheme that channelled most of the aid to the large farms in the best agricultural regions.
Revised: • It would be difﬁcult to support a subsidy scheme that channelled most of the aid to the large farms in the best agricultural regions.

Original: Es w¨are auch problematisch , ein Beihilfesystem zu bef¨urworten  das die meisten Beihilfen in die großen Betriebe in den bestenlandwirtschaftlichen
Revised: Es w¨are auch problematisch, ein Beihilfesystem zu bef¨urworten , das die meisten Beihilfen in die großen Betriebe in den besten

Original:  Gebieten lenkt.
Revised: landwirtschaftlichen Gebieten lenkt.

Original: Multiple meanings of “close”:
Revised: Multiple meanings of “close”:

Original: • This empire did not look for colonies far from home or overseas, like most Western European States, but close by.
Revised: • This empire did not look for colonies far from home or overseas, like most Western European States, but close by.

Original: Dieses

Original:  Reich suchte seine Kolonien nicht weit von zu Hause und in bersee 
Revised: Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee wie die meisten westeurop¨aischen

Original: wie die

Original: meisten westeurop¨aischen

Original: Staaten, sondern in der unmittelbaren Umgebung .
Revised: Staaten, sondern in der unmittelbaren Umgebung .

Original: • In addition, if we are to shut down or refuse investment from every company which may have an association with the arms industry, then we would have to close virtually every American and Japanese software company on the island of Ireland with catastrophic consequences.
Revised: • In addition, if we are to shut down or refuse investment from every company which may have an association with the arms industry, then we would have to close virtually every American and Japanese software company on the island of Ireland with catastrophic consequences.

Original: Wenn wir zudem jedes Unternehmen, das auf irgendeine Weise mit der
Revised: Wenn wir zudem jedes Unternehmen, das auf irgendeine Weise mit der

Original: Rstungsindustrie verbunden ist, schließen oder Investitionen dieser Unternehmen unterbinden
Revised: Rstungsindustrie verbunden ist, schließen oder Investitionen dieser Unternehmen unterbinden

Original: , dann mßten wir so ziemlich alle amerikanischen und japanischen Softwareunternehmen auf der irischen
Revised: , dann mßten wir so ziemlich alle amerikanischen und japanischen Softwareunternehmen auf der irischen

Original: Insel schließen , was katastrophale Auswirkungen h¨atte.
Revised: Insel schließen , was katastrophale Auswirkungen h¨atte.

Original: • On the other hand, the deployment of resources left over in the Structural Funds from the programme planning period 1994 to 1999 is hardly worth considering as the available fundshave already been allocated to speciﬁc measures, in this case in close collaboration with the relevant French authorities.
Revised: • On the other hand, the deployment of resources left over in the Structural Funds from the programme planning period 1994 to 1999 is hardly worth considering as the available funds

Original: Die
Revised: have

Revised: already

Revised: been allocated to speciﬁc measures, in this case in close collaboration with the relevant French authorities

Original:  Verwendung verbliebener Mittel der Strukturfonds ausdem Programmpla-
Revised: Die Verwendung verbliebener Mittel der Strukturfonds aus

Original: nungszeitraum 1994 bis 1999 ist dagegen kaum in Erw¨agung zu 
Revised: dem Programmplanungszeitraum 1994 bis 1999 ist dagegen kaum in Erw¨agung zu ziehen, da die verfgbaren

Original: ziehen

Original: da

Original: die verfgbaren Mittel bereits bestimmten Maßnahmen zugewiesen sind, und zwar im konkreten
Revised: Mittel bereits bestimmten Maßnahmen zugewiesen sind, und zwar im konkreten

Original: Fall im engen Zusammenwirken mit den zust ¨ andigen franz ¨ osischen Beh
Revised: Fall im engen Zusammenwirken mit den zust¨andigen franz¨osischen Beh¨orden.

Original: ¨ orden • This is particularly justiﬁed given that, as already stated, many Member States have very close relations with Djibouti.
Revised: • This is particularly justiﬁed given that, as already stated, many Member States have very close relations with Djibouti.

Original: Zumal, wie erw¨ahnt, viele Mitgliedstaaten sehr enge Beziehungen zu Dschibuti unterhalten.
Revised: Zumal, wie erw¨ahnt, viele Mitgliedstaaten sehr enge Beziehungen zu Dschibuti unterhalten.

Original: • Mr President, it is regrettable that, at the close of the 20th century, a century symbolised so positively by the peaceful women’s revolution, there are still countries, such as Kuwait and Afghanistan, where half the population, women that is, is still denied fundamental human rights.
Revised: • Mr President, it is regrettable that, at the close of the 20th century, a century symbolised so positively by the peaceful women’s revolution, there are still countries, such as Kuwait and Afghanistan, where half the population, women that is, is still denied fundamental human rights.

Original: Herr Pr¨asident!
Revised: Herr Pr¨asident!

Original: Es ist wirklich bedauerlich, daß es am Ende des 20.
Revised: Es ist wirklich bedauerlich, daß es am Ende des 20.

Original: Jahrhunderts, eines so positiv von der friedlichen Revolution der Frauen gepr¨agten Jahrhunderts, noch immer
Revised: Jahrhunderts, eines so positiv von der friedlichen Revolution der Frauen gepr¨agten Jahrhunderts, noch immer

Original: L ¨ ander wie Kuwait und Afghanistan gibt, in denen der H ¨ alfte der Bev ¨ olkerung, den Frauen, die elementaren Menschenrechte verweigert werden.
Revised: L ¨ ander wie Kuwait und Afghanistan gibt, in denen der H ¨ alfte der Bev ¨ olkerung, den Frauen, die elementaren Menschenrechte verweigert werden.