Original: C ROSS - LINGUAL A LIGNMENT VS J OINT T RAINING : A C OMPARATIVE S TUDY AND A S IMPLE U NIFIED F RAMEWORK
Revised: C ROSS - LINGUAL A LIGNMENT VS J OINT T RAINING : A C OMPARATIVE S TUDY AND A S IMPLE U NIFIED F RAMEWORK

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks.
Revised: Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks.

Original: There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns uniﬁed multilingual representations using monolingual and cross-lingual objectives jointly.
Revised: There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns uniﬁed multilingual representations using monolingual and cross-lingual objectives jointly.

Original: In this paper, we ﬁrst conduct direct comparisons of representations learned using both of these methods across diverse crosslingual tasks.
Revised: In this paper, we ﬁrst conduct direct comparisons of representations learned using both of these methods across diverse crosslingual tasks.

Original: Our empirical results reveal a set of pros and cons for both methods and show that the relative performance of alignment versus joint training is task-dependent.
Revised: Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is taskdependent.

Original: Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches.
Revised: Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches.

Original: Extensive experimentson various tasks demonstrate that our proposed framework alleviates limitations of both approaches and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark.
Revised: Extensive experiments demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark.

Original: We further show that our proposed framework can generalize to contextualized representations  and achieves state-of-the-art results on the CoNLL cross-lingual NER benchmark.
Revised: We further show that this framework can generalize to contextualized representations such as Multilingual BERT, and produces state-of-the-art results on the CoNLL cross-lingual NER benchmark.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Continuous word representations (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) have become ubiquitous across a wide range of NLP tasks.
Revised: Continuous word representations (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) have become ubiquitous across a wide range of NLP tasks.

Original: In particular, methods for crosslingual word embeddings (CLWE) have proven a powerful tool for cross-lingual transfer for downstream tasks, such as text classiﬁcation (Klementiev et al., 2012a), dependency parsing (Ahmad et al., 2019), named entity recognition (NER)
Revised: In particular, methods for crosslingual word embeddings (CLWE) have proven a powerful tool for cross-lingual transfer for downstream tasks, such as text classiﬁcation (Klementiev et al., 2012), dependency parsing (Ahmad et al., 2019), named entity recognition (NER)

Original: (Xie et al., 2018;
Revised: (Xie et al., 2018;

Original: Chen et al., 2019), natural language inference (Conneau et al., 2018b), language modeling (Adams et al., 2017), and machine translation (MT) (Zou et al., 2013; Lample et al., 2018a; Artetxe et al., 2018b; Lample et al., 2018b).
Revised: Chen et al., 2019), natural language inference (Conneau et al., 2018b), language modeling (Adams et al., 2017), and machine translation (MT) (Zou et al., 2013; Lample et al., 2018a; Artetxe et al., 2018b; Lample et al., 2018b).

Original: The goal of these CLWE methods is to learn embeddings in a shared vector space for two or more languages.
Revised: The goal of these CLWE methods is to learn embeddings in a shared vector space for two or more languages.

Original: There are two main paradigms for learning CLWE: cross-lingual alignment and joint training .
Revised: There are two main paradigms for learning CLWE: cross-lingual alignment and joint training .

Original: The most successful approach has been the cross-lingual embedding alignment method (Mikolov et al., 2013b), which relies on the assumption that monolingually-trained continuous word embedding spaces share similar structure across different languages.
Revised: The most successful approach has been the cross-lingual embedding alignment method (Mikolov et al., 2013b), which relies on the assumption that monolingually-trained continuous word embedding spaces share similar structure across different languages.

Original: The underlying idea is to ﬁrst independently train embeddings in different languages using monolingual corpora alone, and then learn a mapping to align them to a shared vector space.
Revised: The underlying idea is to ﬁrst independently train embeddings in different languages using monolingual corpora alone, and then learn a mapping to align them to a shared vector space.

Original: Such a mapping can be trained in a supervised fashion using parallel resources such as bilingual lexicons (Xing et al., 2015; Smith et al., 2017;
Revised: Such a mapping can be trained in a supervised fashion using parallel resources such as bilingual lexicons (Xing et al., 2015; Smith et al., 2017;

Original: Joulin et al., 2018b; Jawanpuria et al., 2019), or even in an unsupervised 2 manner based on distribution matching (Zhang et al., 2017a; Conneau et al., 2018a; Artetxe et al., 2018a;
Revised: Joulin et al., 2018b; Jawanpuria et al., 2019), or even in an unsupervised 2 manner based on distribution matching (Zhang et al., 2017a; Conneau et al., 2018a; Artetxe et al., 2018a;

Original: Zhou et al., 2019).
Revised: Zhou et al., 2019).

Original: Recently, it has been shown that alignment methods can also be effectively applied to contextualized word representations (Schuster et al., 2019; Aldarmaki & Diab, 2019).
Revised: Recently, it has been shown that alignment methods can also be effectively applied to contextualized word representations (Schuster et al., 2019; Aldarmaki & Diab, 2019).

Original: Another successful line of research for CLWE considers joint training methods, which optimize a monolingual objective predicting the context of a word in a monolingual corpus along with either a hard or soft cross-lingual constraint.
Revised: Another successful line of research for CLWE considers joint training methods, which optimize a monolingual objective predicting the context of a word in a monolingual corpus along with either a hard or soft cross-lingual constraint.

Original: Similar to alignment methods, some early works rely on bilingual dictionaries (Ammar et al., 2016;
Revised: Similar to alignment methods, some early works rely on bilingual dictionaries (Ammar et al., 2016;

Original: Duong et al., 2016) or parallel corpora (Luong et al., 2015; Gouws et al., 2015) for direct supervision.
Revised: Duong et al., 2016) or parallel corpora (Luong et al., 2015; Gouws et al., 2015) for direct supervision.

Original: More recently, a seemingly naive unsupervised joint training approach has received growing attention due to its simplicity and effectiveness.
Revised: More recently, a seemingly naive unsupervised joint training approach has received growing attention due to its simplicity and effectiveness.

Original: In particular, Lample et al.
Revised: In particular, Lample et al.

Original: (2018b) reports that simply training embeddings on concatenated monolingual corpora of two related languages using a shared vocabulary without any cross-lingual resources is able to produce higher accuracy than the more sophisticated alignment methods on unsupervised MT tasks.
Revised: (2018b) reports that simply training embeddings on concatenated monolingual corpora of two related languages using a shared vocabulary without any cross-lingual resources is able to produce higher accuracy than the more sophisticated alignment methods on unsupervised MT tasks.

Original: Besides, for contextualized representations, unsupervised multilingual language model pretraining using a shared vocabulary has produced state-of-the-art results on multiple benchmarks 3 (Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019).
Revised: Besides, for contextualized representations, unsupervised multilingual language model pretraining using a shared vocabulary has produced state-of-the-art results on multiple benchmarks(Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019).

Original: Despite a large amount of research on both alignment and joint training, previous work has neither performed a systematic comparison between the two, analyzed their pros and cons, nor elucidated when we may prefer one method over the other.
Revised: Despite a large amount of research on both alignment and joint training, previous work has neither performed a systematic comparison between the two, analyzed their pros and cons, nor elucidated when we may prefer one method over the other.

Original: Particularly, it’s natural to ask: (1) Does the phenomenon reported in Lample et al.
Revised: Particularly, it’s natural to ask: (1) Does the phenomenon reported in Lample et al.

Original: (2018b) extend to other cross-lingual tasks?
Revised: (2018b) extend to other cross-lingual tasks?

Original: (2) Can we employ alignment methods to further improvetheir proposed unsupervised joint training?
Revised: (2) Can we employ alignment methods to further improve unsupervised joint training?

Original: If so, how would such a framework compare to supervised joint training methods that exploit equivalent resources?
Revised: If so, how would such a framework compare to supervised joint training methods that exploit equivalent resources, i.e., bilingual dictionaries?

Original: (4) And lastly, can this framework generalize to contextualized representations?
Revised: (4) And lastly, can this framework generalize to contextualized representations?

Original: In this work, we attempt to address these questions.
Revised: In this work, we attempt to address these questions.

Original: Speciﬁcally, we ﬁrst evaluate and compare alignment versus joint training methods across three diverse tasks: BLI, cross-lingual NER, and unsupervised MT.
Revised: Speciﬁcally, we ﬁrst evaluate and compare alignment versus joint training methods across three diverse tasks: BLI, cross-lingual NER, and unsupervised MT.

Original: We seek to characterize the conditions under which one approach outperforms the other, and glean insight on the reasons behind these differences.
Revised: We seek to characterize the conditions under which one approach outperforms the other, and glean insight on the reasons behind these differences.

Original: Based on our analysis, we further propose a simple, novel, and highly generic framework that uses unsupervised joint training as initialization and alignment as reﬁnement to combine both paradigms.
Revised: Based on our analysis, we further propose a simple, novel, and highly generic framework that uses unsupervised joint training as initialization and alignment as reﬁnement to combine both paradigms.

Original: Our experiments demonstrate that our framework improves over both alignment and joint training baselines, and outperforms existing methods on the MUSE BLI benchmark.
Revised: Our experiments demonstrate that our framework improves over both alignment and joint training baselines, and outperforms existing methods on the MUSE BLI benchmark.

Original: Moreover, we show that our framework can generalize to contextualized representations, producing state-of-the-art results on the CoNLL cross-lingual NER benchmark.
Revised: Moreover, we show that our framework can generalize to contextualized representations such as Multilingual BERT, producing state-of-the-art results on the CoNLL cross-lingual NER benchmark.

Original: To the best of our knowledge, this is the ﬁrst framework that combines previously mutually-exclusive alignment and joint training methods.
Revised: To the best of our knowledge, this is the ﬁrst framework that combines previously mutually-exclusive alignment and joint training methods.

Original:  B ACKGROUND : C ROSS - LINGUAL R EPRESENTATIONS
Revised:  B ACKGROUND : C ROSS - LINGUAL R EPRESENTATIONS

Original: Notation.
Revised: Notation.

Original: We assume we have two different languages { L 1 , L 2 } and access to their corresponding training corpora.
Revised: We assume we have two different languages { L 1 , L 2 } and access to their corresponding training corpora.

Original: We use V L i = { w jL i } n Li j =1 to denote the vocabulary set of the i th language where each w jL i represents a unique token, such as a word or subword.
Revised: We use V L i = { w jL i } n Li j =1 to denote the vocabulary set of the i th language where each w jL i represents a unique token, such as a word or subword.

Original: The goal is to learn a set of embeddings E = { x j } mj =1 , with x j ∈ R d , in a shared vector space, where each token w jL i is mapped to a vector in E .
Revised: The goal is to learn a set of embeddings E = { x j } mj =1 , with x j ∈ R d , in a shared vector space, where each token w jL i is mapped to a vector in E .

Original: Ideally, these vectorial representations should have similar values for tokens with similar meanings or syntactic properties, so they can better facilitate cross-lingual transfer.
Revised: Ideally, these vectorial representations should have similar values for tokens with similar meanings or syntactic properties, so they can better facilitate cross-lingual transfer.

Original: 2. A LIGNMENT M ETHODS
Revised: 2. A LIGNMENT M ETHODS

Original: Given the notation, alignment methods consist of the following steps:
Revised: Given the notation, alignment methods consist of the following steps:

Original: Step1: Train an embedding set E 0 ={+
+}
Revised: Step Train an embedding set E {+
+}

Original:  E L 1 ∪ E L 2 , where each subset E L
Revised: = E L 1 ∪ E L 2 , where each subset E L

Original: i
Revised: i

Original: =
Revised: =

Original: { x jL i } n Li j =1 is trained independently using the i th language corpus and contains an embedding x jL i for each token w jL i .
Revised: { x jL i } n Li j =1 is trained independently using the i th language corpus and contains an embedding x jL i for each token w jL i .

Original: Step 2: Obtain a seed dictionary D
Revised: Step 2: Obtain a seed dictionary D

Original: ={ ( w iL1 , w jL 2 ) } 
Revised: = w iL w jL 2 ) } Kk = 1 , either

Original: Kk =1 , either provided or learnt unsupervised.
Revised: provided or learnt unsupervised.

Original: Step 3: Learn a projection matrix W ∈ R d × d based on D , resulting in a ﬁnal embedding set E A 
Revised: Step 3: Learn a projection matrix W ∈ R d × d based on D , resulting in a ﬁnal embedding set E A =

Original: = ( W · E L 1 ) ∪ E L 2 in a shared vector space.
Revised: ( W · E L 1 ) ∪ E L 2 in a shared vector space.

Original: To ﬁnd the optimal projection matrix W , Mikolov et al.
Revised: To ﬁnd the optimal projection matrix W , Mikolov et al.

Original: (2013b) proposed to solve the following optimization problem:
Revised: (2013b) proposed to solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: (1) where X L 1 and X L 2 are matrices of size d × K containing embeddings of the words in D .
Revised: where X L 1 and X L 2 are matrices of size d × K containing embeddings of the words in D .

Original: Xing et al.
Revised: Xing et al.

Original: (2015) later showed further improvement could be achieved by restricting W to an orthogonal matrix, which turns the Eq.(1) into the Procrustes problem with the following closed form solution:
Revised: (2015) later showed further improvement could be achieved by restricting W to an orthogonal matrix, which turns the Eq.(1) into the Procrustes problem with the following closed form solution:

Original: [Equation]
Revised: [Equation]

Original: where W ∗ denotes the optimal solution and SVD( · ) stands for the singular value decomposition.
Revised: where W ∗ denotes the optimal solution and SVD( · ) stands for the singular value decomposition.

Original: As surveyed in Section 5, different methods (Smith et al., 2017; Conneau et al., 2018a; Joulin et al., 2018b; Artetxe et al., 2018a) differ in the way how they obtain the dictionary D and how they solve for W in step 3.
Revised: As surveyed in Section 5, different methods (Smith et al., 2017; Conneau et al., 2018a; Joulin et al., 2018b; Artetxe et al., 2018a) differ in the way how they obtain the dictionary D and how they solve for W in step 3.

Original: However, most of them still involve solving the Eq.(2) as a crucial step.
Revised: However, most of them still involve solving the Eq.(2) as a crucial step.

Original: 2. J OINT T RAINING M ETHODS
Revised: 2. J OINT T RAINING M ETHODS

Original: Joint training methods in general have the following objective:
Revised: Joint training methods in general have the following objective:

Original: [Equation]
Revised: [Equation]

Original: where L 1 and L 2 are monolingual objectives and R ( L 1 , L 2 ) is a cross-lingual regularization term.
Revised: where L 1 and L 2 are monolingual objectives and R ( L 1 , L 2 ) is a cross-lingual regularization term.

Original: For example, Klementiev et al.
Revised: For example, Klementiev et al.

Original: (2012b) uses language modeling objectives for L 1 and L 2 .
Revised: (2012) use language modeling objectives for L 1 and L 2 .

Original: The term R ( L 1 , L 2 ) encourages alignment of representations of words that are translations.
Revised: The term R ( L 1 , L 2 ) encourages alignment of representations of words that are translations.

Original: Training an embedding set E J = E L 1 ∪ E L 2 is usually done by directly optimizing L J .
Revised: Training an embedding set E J = E L 1 ∪ E L 2 is usually done by directly optimizing L J .

Original: While supervised joint training requires access to parallel resources, recent studies (Lample et al., 2018b; Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019) have suggested that unsupervised joint training without such resources are also effective.
Revised: While supervised joint training requires access to parallel resources, recent studies (Lample et al., 2018b; Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019) have suggested that unsupervised joint training without such resources is also effective.

Original: Speciﬁcally, they show that the cross-lingual regularization term R ( L 1 , L 2 ) does not require direct cross-lingual supervision to achieve highly competitive results.
Revised: Speciﬁcally, they show that the cross-lingual regularization term R ( L 1 , L 2 ) does not require direct cross-lingual supervision to achieve highly competitive results.

Original: This is because the shared words between L 1 and L 2 can serve implicitly as translations by sharing their embeddings to ensure that representations of different languages lie in a shared space.
Revised: This is because the shared words between L 1 and L 2 can serve implicitly as anchors by sharing their embeddings to ensure that representations of different languages lie in a shared space.

Original: Using our notation, the unsupervised joint training approach takes the following steps:
Revised: Using our notation, the unsupervised joint training approach takes the following steps:

Original: Step 1: Construct a joint vocabulary V J = V L 1 ∪ V L 2 that is shared across two languages.
Revised: Step 1: Construct a joint vocabulary V J = V L 1 ∪ V L 2 that is shared across two languages.

Original: Step 2: Concatenate the two training corpora and learn an embedding set E J corresponding to V J .
Revised: Step 2: Concatenate the two training corpora and learn an embedding set E J corresponding to V J .

Original: The joint vocabulary is composed of three disjoint sets: V 1 J , V 2 J , V sJ , where V sJ = V L 1 ∩ V L 2 is the shared vocabulary set and V iJ is the set of tokens that appear in the i th language only.
Revised: The joint vocabulary is composed of three disjoint sets: V 1 J , V 2 J , V sJ , where V sJ = V L 1 ∩ V L 2 is the shared vocabulary set and V iJ is the set of tokens that appear in the i th language only.

Original: Note that a key difference of existing supervised joint training methods is that embeddings corresponding to V sJ are not shared between E L 1 and E L 2 , meaning that they are disjoint as in alignment methods.
Revised: Note that a key difference of existing supervised joint training methods is that embeddings corresponding to V sJ are not shared between E L 1 and E L 2 , meaning that they are disjoint, as in alignment methods.

Original:  D ISCUSSION
Revised: 2. D ISCUSSION

Original: While alignment methods have had great success, there are still some critical downsides, among which we stress the following points:
Revised: While alignment methods have had great success, there are still some critical downsides, among which we stress the following points:

Original: While recent studies in unsupervised joint training have suggested the potential beneﬁts of word sharing, alignment methods rely on two disjoint sets of embeddings.
Revised: While recent studies in unsupervised joint training have suggested the potential beneﬁts of word sharing, alignment methods rely on two disjoint sets of embeddings.

Original: Along with some possible loss of information due to no sharing, one consequence is that ﬁnetuning the aligned embeddings on downstream tasks may be sub-optimal due to the lack of crosslingual constraints at the ﬁnetuning stage, whereas shared words can fulﬁll this role in jointly trained models.
Revised: Along with some possible loss of information due to no sharing, one consequence is that ﬁnetuning the aligned embeddings on downstream tasks may be sub-optimal due to the lack of crosslingual constraints at the ﬁnetuning stage, whereas shared words can fulﬁll this role in jointly trained models.

Original: A key assumption of alignment methods is the isomorphism of monolingual embedding spaces.
Revised: A key assumption of alignment methods is the isomorphism of monolingual embedding spaces.

Original: However, some recent papers have challenged this assumption, showing that it does not hold for many language pairs (Søgaard et al., 2018; Patra et al., 2019).
Revised: However, some recent papers have challenged this assumption, showing that it does not hold for many language pairs (Søgaard et al., 2018; Patra et al., 2019).

Original: Also notably, Ormazabal et al.
Revised: Also notably, Ormazabal et al.

Original: (2019) suggests that this limitation results from the fact that the two sets of monolingual embeddings are independently trained.
Revised: (2019) suggests that this limitation results from the fact that the two sets of monolingual embeddings are independently trained.

Original: On the other hand, the unsupervised joint training method is much simpler and doesn’t share these disadvantages with the alignment methods, but there are also some key limitations:
Revised: On the other hand, the unsupervised joint training method is much simpler and doesn’t share these disadvantages with the alignment methods, but there are also some key limitations:

Original:  It assumes that all shared words across two languages serve implicitly as translations and thus need not be aligned to other words.
Revised: [Equation] It assumes that all shared words across two languages serve implicitly as anchors and thus need not be aligned to other words.

Original: Nonetheless, this assumption is not always true, leading to misalignment.
Revised: Nonetheless, this assumption is not always true, leading to misalignment.

Original: For example, the English word “the” will most likely also appear 
Revised: For example, the English word “the” will most likely also appear in the training corpus of Spanish, but preferably it should be paired with Spanish words such as “el” and “la” instead of itself. We refer to this problem as oversharing. It does not utilize any explicit form of seed dictionary as in alignment methods, resulting in potentially less accurate alignments, especially for words that are not shared.

Original: Lastly, while the supervised joint training approach does not have the same issues of unsupervised joint training, it shares limitation 1 of the alignment methods.
Revised: Lastly, while the supervised joint training approach does not have the same issues of unsupervised joint training, it shares limitation 1 of the alignment methods.

Original: We empirically compare both joint training and alignment approaches in Section 4 and shed light on some of these pros and cons for both paradigms (See Section 4.3.1).
Revised: We empirically compare both joint training and alignment approaches in Section 4 and shed light on some of these pros and cons for both paradigms (See Section 4.3.1).

Original:  P ROPOSED F RAMEWORK
Revised:  P ROPOSED F RAMEWORK

Original: Motivated by the pros and cons of both paradigms, we propose a uniﬁed framework that ﬁrst uses unsupervised joint training as a coarse initialization and then applies alignment methods for reﬁnement, as demonstrated in Figure 1.
Revised: Motivated by the pros and cons of both paradigms, we propose a uniﬁed framework that ﬁrst uses unsupervised joint training as a coarse initialization and then applies alignment methods for reﬁnement, as demonstrated in Figure 1.

Original: Speciﬁcally, we ﬁrst build a single set of embeddings with a shared vocabulary through unsupervised joint training, so as to alleviate the limitations of alignment methods.
Revised: Speciﬁcally, we ﬁrst build a single set of embeddings with a shared vocabulary through unsupervised joint training, so as to alleviate the limitations of alignment methods.

Original: Next, we use a vocabulary reallocation technique to mitigate oversharing, before ﬁnally resorting back to alignment methods to further improve the embeddings’ quality. 
Revised: Next, we use a vocabulary reallocation technique to mitigate oversharing, before ﬁnally resorting back to alignment methods to further improve the embeddings’ quality. Lastly, we show that this framework can generalize to contextualized representations.

Original: 3. U NIFYING A LIGNMENT WITH J OINT T RAINING
Revised: 3. U NIFYING A LIGNMENT WITH J OINT T RAINING

Original: Our proposed framework mainly involves three components and we discuss each of them as follows.
Revised: Our proposed framework mainly involves three components and we discuss each of them as follows.

Original: Joint Initialization.
Revised: Joint Initialization.

Original: We use unsupervised joint training (Lample et al., 2018b) to train the initial
Revised: We use unsupervised joint training (Lample et al., 2018b) to train the initial

Original: CLWE.
Revised: CLWE.

Original: As described in Section 2.2, we ﬁrst obtain a joint vocabulary V J and train its corresponding set of embeddings E J on the concatenated corpora of two languages.
Revised: As described in Section 2.2, we ﬁrst obtain a joint vocabulary V J and train its corresponding set of embeddings E J on the concatenated corpora of two languages.

Original: This allows us to obtain a single set of embeddings that maximizes sharing across two languages.
Revised: This allows us to obtain a single set of embeddings that maximizes sharing across two languages.

Original: To train embeddings, we used fastText  (Bojanowski et al., 2017) in all our experiments for both word and subword tokens.
Revised: To train embeddings, we used fastText 4 (Bojanowski et al., 2017) in all our experiments for both word and subword tokens.

Original: Vocabulary Reallocation
Revised: Vocabulary Reallocation.

Original: As discussed in Secition 2.3, a key issue of unsupervised joint training is oversharing, which prohibits further reﬁnement as shown in Figure 1.
Revised: As discussed in Section 2.3, a key issue of unsupervised joint training is oversharing, which prohibits further reﬁnement as shown in Figure 1.

Original: To alleviate this drawback, we attempt to “unshare” some of the overshared words, so their embeddings can be better aligned in the next step.
Revised: To alleviate this drawback, we attempt to “unshare” some of the overshared words, so their embeddings can be better aligned in the next step.

Original: Particularly, we perform a vocabulary reallocation step such that words appearing mostly exclusively in the i th language are reallocated from the shared vocabulary V sJ to V iJ , whereas words that appear similarly frequent in both languages stay still in V sJ .
Revised: Particularly, we perform a vocabulary reallocation step such that words appearing mostly exclusively in the i th language are reallocated from the shared vocabulary V sJ to V iJ , whereas words that appear similarly frequent in both languages stay still in V sJ .

Original: Formally, for each token w in the shared vocabulary V sJ , we use the ratio of counts within each language to determine whether it belongs to the shared vocabulary:
Revised: Formally, for each token w in the shared vocabulary V sJ , we use the ratio of counts within each language to determine whether it belongs to the shared vocabulary:

Original: [Equation]
Revised: [Equation]

Original: where C L i ( w ) is the count of w in the training corpus of the i th language and
Revised: where C L i ( w ) is the count of w in the training corpus of the i th language and

Original: T L i = (cid:80) w C L
Revised: T L i = (cid:80) w C L

Original: i ( w ) is the total number of tokens.
Revised: i ( w ) is the total number of tokens.

Original: The token w is allocated to the shared vocabulary if
Revised: The token w is allocated to the shared vocabulary if

Original: [Equation]
Revised: [Equation]

Original: where γ is a hyper-parameter.
Revised: where γ is a hyper-parameter.

Original: Otherwise, we put w into either V 1 J or V 2 J , where it appears mostly frequent.
Revised: Otherwise, we put w into either V 1 J or V 2 J , where it appears mostly frequent.

Original: The above process generates three new disjoint vocabulary sets V 1 (cid:48)
Revised: The above process generates three new disjoint vocabulary sets V 1 (cid:48)

Original: J , V 2 (cid:48)
Revised: J , V 2 (cid:48)

Original: J , V s (cid:48)
Revised: J , V s (cid:48)

Original: J and their corresponding embeddings E 1 (cid:48)
Revised: J and their corresponding embeddings E 1 (cid:48)

Original: J , E 2 (cid:48)
Revised: J , E 2 (cid:48)

Original: J , E s (cid:48)
Revised: J , E s (cid:48)

Original: J that are used thereafter.
Revised: J that are used thereafter.

Original: Note that, V (cid:48)
Revised: Note that, V (cid:48)

Original: J = V J and E (cid:48)
Revised: J = V J and E (cid:48)

Original: J = E J .
Revised: J = E J .

Original: Alignment Reﬁnement.
Revised: Alignment Reﬁnement.

Original: The unsupervised joint training method does not explicitly utilize any dictionary or form of alignment.
Revised: The unsupervised joint training method does not explicitly utilize any dictionary or form of alignment.

Original: Thus, the resulting embedding set is coarse and ill-aligned in the shared vector space, as demonstrated in Figure 1.
Revised: Thus, the resulting embedding set is coarse and ill-aligned in the shared vector space, as demonstrated in Figure 1.

Original: As a ﬁnal reﬁnement step, we utilize any off-the-shelf alignment method to reﬁne alignments across the non-sharing embedding sets, i.e. mapping E 1 (cid:48)
Revised: As a ﬁnal reﬁnement step, we utilize any off-the-shelf alignment method to reﬁne alignments across the non-sharing embedding sets, i.e. mapping E 1 (cid:48)

Original: J to E 2 (cid:48)
Revised: J to E 2 (cid:48)

Original: J and leaving E s (cid:48)
Revised: J and leaving E s (cid:48)

Original: J untouched.
Revised: J untouched.

Original: This step could be conducted by either supervised or unsupervised alignment method and we compare both in our experiments.
Revised: This step could be conducted by either supervised or unsupervised alignment method and we compare both in our experiments.

Original: 3. E XTENSION TO C ONTEXTUALIZED R EPRESENTATIONS
Revised: 3. E XTENSION TO C ONTEXTUALIZED R EPRESENTATIONS

Original: As our framework is highly generic and applicable to any alignment and unsupervised joint training methods, it can naturally generalize to contextualized word representations by aligning the ﬁxed outputs of a multilingual encoder such as multilingual BERT (M-BERT) (Devlin et al., 2019).
Revised: As our framework is highly generic and applicable to any alignment and unsupervised joint training methods, it can naturally generalize to contextualized word representations by aligning the ﬁxed outputs of a multilingual encoder such as multilingual BERT (M-BERT) (Devlin et al., 2019).

Original: While our vocab reallocation technique is no longer necessary as contextualized representations are dependent on context and thus dynamic, we can still apply alignment reﬁnement on extracted contextualized features for further improvement.
Revised: While our vocab reallocation technique is no longer necessary as contextualized representations are dependent on context and thus dynamic, we can still apply alignment reﬁnement on extracted contextualized features for further improvement.

Original: For instance, as proposed by Aldarmaki & Diab (2019), one method to perform alignment on contextualized representations is to ﬁrst use word alignment pairs extracted from parallel corpora as a dictionary, learn an alignment matrix W based on it, and apply
Revised: For instance, as proposed by Aldarmaki & Diab (2019), one method to perform alignment on contextualized representations is to ﬁrst use word alignment pairs extracted from parallel corpora as a dictionary, learn an alignment matrix W based on it, and apply

Original: W back to the extracted representations.
Revised: W back to the extracted representations.

Original: To obtain W , we can solve Eq.
Revised: To obtain W , we can solve Eq.

Original: ( 1) as described in Section 2.1, where the embedding matrices X L 1 and X L 2 now contain contextualized representations of aligned word pairs.
Revised: ( 1) as described in Section 2.1, where the embedding matrices X L 1 and X L 2 now contain contextualized representations of aligned word pairs.

Original: Note that this method is applicable to ﬁxed representations but not ﬁnetuning.
Revised: Note that this method is applicable to ﬁxed representations but not ﬁnetuning.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the proposed approach and compare with alignment and joint training methods on three
Revised: We evaluate the proposed approach and compare with alignment and joint training methods on three

Original: NLP benchmarks.
Revised: NLP benchmarks.

Original: This evaluation aims to: (1) systematically compare alignment vs. joint training paradigms and reveal their pros and cons discussed in Section 2.3, (2) show that the proposed framework can effectively alleviate limitations of both alignment and joint training, and (3) demonstrate the effectiveness of the proposed framework in both non-contextualized and contextualized settings.
Revised: This evaluation aims to: (1) systematically compare alignment vs. joint training paradigms and reveal their pros and cons discussed in Section 2.3, (2) show that the proposed framework can effectively alleviate limitations of both alignment and joint training, and (3) demonstrate the effectiveness of the proposed framework in both non-contextualized and contextualized settings.

Original: 4. E VALUATION T ASKS
Revised: 4. E VALUATION T ASKS

Original: Bilingual Lexicon Induction (BLI)
Revised: Bilingual Lexicon Induction (BLI)

Original: This task has been the de facto evaluation task for CLWE methods.
Revised: This task has been the de facto evaluation task for CLWE methods.

Original: It considers the problem of retrieving the target language translations of source langauge words.
Revised: It considers the problem of retrieving the target language translations of source langauge words.

Original: We use bilingual dictionaries complied by Conneau et al.
Revised: We use bilingual dictionaries complied by Conneau et al.

Original: (2018a)
Revised: (2018a)

Original: and test on six diverse language pairs, including Chinese and Russian, which use a different writing script than English.
Revised: and test on six diverse language pairs, including Chinese and Russian, which use a different writing script than English.

Original: Each test set consists of 1500 queries and we report precision at 1 scores (P@1), following standard evaluation practices (Conneau et al., 2018a; Glavas et al., 2019).
Revised: Each test set consists of 1500 queries and we report precision at 1 scores (P@1), following standard evaluation practices (Conneau et al., 2018a; Glavas et al., 2019).

Original: Name Entity Recognition (NER)
Revised: Name Entity Recognition (NER)

Original: We also evaluate our proposed framework on cross-lingual NER, a sequence labeling task, where we assign a label to each token in a sequence.
Revised: We also evaluate our proposed framework on cross-lingual NER, a sequence labeling task, where we assign a label to each token in a sequence.

Original: We evaluate both non-contextualized and contextualized word representations on the CoNLL 2002 and 2003 benchmarks (Tjong Kim Sang, 2002; Tjong Kim Sang & De Meulder, 2003) , which contain 4 European languages.
Revised: We evaluate both en-es es-en en-fr fr-en en-de de-en en-it it-en en-ru ru-en en-zh zh-en avg AlignmentMethods (1)MUSE(Conneauetal.,2018a) 81. [Table] [Table] [Table]

Original: To measure the quality of CLWE, we perform zero-shot cross-lingual classiﬁcation, where we train a model on English and directly apply it to each of the other 3 languages.
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: (Duongetal.,2016)
Revised: [Table] [Table] [Table] (2)VECMAP(Artetxeetal.,2018a) 82.

Original: [Table]
Revised: [Table]

Original: 65.2 JointAlignFramework
Revised: [Table]

Original: Joint Align(w/oAR) 56.
Revised: [Table] [Table] [Table] [Table]

Original: 42.6 Joint Align+MUSE 81.
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Revised: - (3)DeMa-BWE(Zhouetal.

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Revised: )Procrustes(Smithetal.

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: GeoMM
Revised: )GeoMM(Jawanpuriaetal.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Revised: (6)RCSLS(Joulinetal.,2018b) 84.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: 72.4 (7)RCSLS+IN(Zhangetal.,2019) 83.

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: - - JointTraingMethods

Original: [Table]
Revised: [Table]

Revised: non-contextualized and contextualized word representations on the CoNLL 2002 and 2003 benchmarks (Tjong Kim Sang, 2002; Tjong Kim Sang & De Meulder, 2003), which contain 4 European languages.

Revised: To measure the quality of CLWE, we perform zero-shot cross-lingual classiﬁcation, where we train a model on English and directly apply it to each of the other 3 languages.

Original: Unsupervised Machine Translation (UMT) Lastly, we test our approach using the unsupervised
Revised: Unsupervised Machine Translation (UMT) Lastly, we test our approach using the unsupervised

Original: MT task, on which the initialization of CLWE plays a crucial role (Lample et al., 2018b).
Revised: MT task, on which the initialization of CLWE plays a crucial role (Lample et al., 2018b).

Original: Note that our purpose here is to directly compare with similar studies in Lample et al.
Revised: Note that our purpose here is to directly compare with similar studies in Lample et al.

Original: (2018b), and
Revised: (2018b), and

Original: thus we follow their settings and consider two language pairs, English-French and English-German, and evaluate on the widely used WMT’14 en-fr and WMT’16 en-de benchmarks.
Revised: thus we follow their settings and consider two language pairs, English-French and English-German, and evaluate on the widely used WMT’14 en-fr and WMT’16 en-de benchmarks.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: For the BLI task, we compare our framework to recent state-of-the-art methods.
Revised: For the BLI task, we compare our framework to recent state-of-the-art methods.

Original: We obtain numbers from the corresponding papers or Zhou et al.
Revised: We obtain numbers from the corresponding papers or Zhou et al.

Original: and use the ofﬁcial tools for MUSE (Conneau et al., 2018a), GeoMM (Jawanpuria et al., 2019) and RCSLS (Joulin et al., 2018b) to obtain missing results.
Revised: and use the ofﬁcial tools for MUSE (Conneau et al., 2018a), GeoMM (Jawanpuria et al., 2019) and RCSLS (Joulin et al., 2018b) to obtain missing results.

Original: We consider the method of Duong et al.
Revised: We consider the method of Duong et al.

Original: (2016) for supervised joint training based on bilingual dictionaries, which is comparable to supervised alignment methods in terms of resources used.
Revised: (2016) for supervised joint training based on bilingual dictionaries, which is comparable to supervised alignment methods in terms of resources used.

Original: For unsupervised joint training, we train uncased joint fastText4 word vectors of dimension 300 on concatenated Wikipedia corpora of each language pair with default parameters.
Revised: For unsupervised joint training, we train uncased joint fastText word vectors of dimension 300 on concatenated Wikipedia corpora of each language pair with default parameters.

Original: The hyperparameter γ is selected from { 0.7, 0.8, 0.9, 0.95 } on validation sets.
Revised: The hyperparameter γ is selected from { 0.7, 0.8, 0.9, 0.95 } on validation sets.

Original: For the alignment reﬁnement step in our proposed framework, we use RCSLS and GeoMM to compare with supervised methods, and MUSE for unsupervised methods.
Revised: For the alignment reﬁnement step in our proposed framework, we use RCSLS and GeoMM to compare with supervised methods, and MUSE for unsupervised methods.

Revised: In addition, we include an additional baseline of joint training, denoted as Joint - Replace , which is identical to unsupervised joint training except that it utilizes a seed dictionary to randomly replace words with their translations in the training corpus.

Original: Following standard practices, we consider the top 200k most frequent words and use the cross-domain similarity local scaling (CSLS) (Conneau et al., 2018a) as the retrieval criteria.
Revised: Following standard practices, we consider the top 200k most frequent words and use the crossdomain similarity local scaling (CSLS) (Conneau et al., 2018a) as the retrieval criteria.

Original: Note that a concurrent work (Artetxe et al., 2019) proposed a new retrieval method based on MT systems and produced state-of-the-art results.
Revised: Note that a concurrent work (Artetxe et al., 2019) proposed a new retrieval method based on MT systems and produced state-of-the-art results.

Original: Although their method is applicable to our framework, it has high computational costs and is out of the scope of this work.
Revised: Although their method is applicable to our framework, it has high computational cost and is out of the scope of this work.

Original: For the NER task: (1) For non-contextualized representations, we train embeddings the same way as in the BLI task and use a vanilla Bi-LSTM-CRF model (Lample et al.,2016; Ma & Hovy, 2016).
Revised: For the NER task: (1) For non-contextualized representations, we train embeddings the same way as in the BLI task and use a vanilla Bi-LSTM-CRF model (Lample et al., 2016).

Original: For all alignment steps, we apply the supervised Procrustes method using dictionaries from the MUSE library for simplicity.
Revised: For all alignment steps, we apply the supervised Procrustes method using dictionaries from the MUSE library for simplicity.

Original: (2) For contextualized representations, we consider two models, M-BERT and XLM (Lample & Conneau, 2019), one unsupervised and one supervised joint training model, respectively.

Original: We try our framework on M-BERT, applying alignment reﬁnement as described in

Original: Section 3.2.

Original: We compare our proposed framework to both ﬁne-tuning and feature extraction.

Original: To use the extracted features, we employ a task-speciﬁc model consisting of 2 Bi-LSTM layers with a total dimension of 768 and a CRF layer.

Original: For ﬁnetuning, we add a softmax layer on top.

Original: To align the contextualized embeddings, we use 30k parallel sentences from the Europarl corpus and follow the procedure of Section 3.2.

Original: Note that, instead of BPE, we use word alignments on parallel data and use average BPE embeddings corresponding to each word to learn the alignment matrix.

Original: We use the sum of the last 4 layer outputs as the extracted features, and learn one matrix for each layer.

Original: For both models, we only predict the label for the ﬁrst subword token corresponding to its original word.

Original:  en-es es-en en-fr fr-en en-de de-en en-it it-en en-ru ru-en en-zh zh-en avg Unsupervised MUSE(Conneauetal. ,2018a) 77.
Revised: (2) For contextualized representations, we use M-BERT, an unsupervised joint training model, as en-es es-en en-fr fr-en en-de de-en en-it it-en en-ru ru-en en-zh zh-en avg Unsupervised (1)MUSE(Conneauetal.,2018a) 77.

Original: 62.5 UnsupervisedJoint 3.
Revised: 62.5 (2)UnsupervisedJoint 3.

Original: 6.4 Joint Align+MUSE 77.
Revised: 6.4 (3)Joint Align+MUSE 77.

Original: 63.7 Supervised RCSLS(Joulinetal. ,2018b) 78.
Revised: 63.7 Supervised (4)RCSLS(Joulinetal.,2018b) 78.

Original: 67.0 SupervisedJoint(Duongetal.,2016)
Revised: 67.0 (5)SupervisedJoint(Duongetal.,2016)

Original: 63.6 Joint Align+RCSLS 82.
Revised: 63.6 (6)Joint Align+RCSLS 82.

Revised: our base model and apply our proposed framework on it by ﬁrst aligning its extracted features and then feeding them to a task-speciﬁc model (M-BERT Feature + Align).

Revised: Speciﬁcally, we use the sum of the last 4 M-BERT layers’ outputs as the extracted features.

Revised: To obtain the alignment matrices, one for each layer, we use 30k parallel sentences from the Europarl corpus for each language pair and follow the procedure of Section 3.2.

Revised: We feed the extracted features as inputs to a task-speciﬁc model with 2 Bi-LSTM layers and a CRF layer (see appendix A).

Revised: We compare our framework to both ﬁnetuning (M-BERT Finetune), which has been studied by previous papers, and feature extraction (M-BERT Feature).

Original:  For the UMT task, we use the exact same  architecture and parameters released by Lample et al.
Revised: Lastly, we also compare against XLM, a supervised joint training model. For the UMT task, we use the exact same data, architecture and parameters released by Lample et al.

Original: (2018b) 5 .
Revised: (2018b) 7 .

Original: We simply use different embeddings  as inputs to the model.
Revised: We simply use different embeddings trained with the same data as inputs to the model.

Original: 4. R ESULTS AND A NALYSIS 4.3. A LIGNMENT VS . J OINT T RAINING
Revised: 4. R ESULTS AND A NALYSIS 4.3. A LIGNMENT VS . J OINT T RAINING

Original: We compare alignment methods with joint training on all three downstream tasks.
Revised: We compare alignment methods with joint training on all three downstream tasks.

Original: As shown in Table 1 and Table 3, we ﬁnd alignment methods signiﬁcantly outperform the joint training approach by a large margin in all language pairs for both BLI and NER.
Revised: As shown in Table 1 and Table 3, we ﬁnd alignment methods signiﬁcantly outperform the joint training approach by a large margin in all language pairs for both BLI and NER.

Original: However, the unsupervised joint training method is superior than its alignment counterpart on the unsupervised MT task as demonstrated in 2(c).
Revised: However, the unsupervised joint training method is superior than its alignment counterpart on the unsupervised MT task as demonstrated in 2(c).

Original: While these results demonstrate that their relative performance is task-dependent, we conduct further analysis to reveal three limitations as discussed in Sec 2.3.
Revised: While these results demonstrate that their relative performance is task-dependent, we conduct further analysis to reveal three limitations as discussed in Sec 2.3.

Original: First, our experiments show that unsupervised joint training fails to generate high-quality alignments due to the lack of  ﬁne-grained seed dictionary as discussed in its limitation 2.On both BLI and NER tasks, alignment methods signiﬁcantly outperform unsupervised joint training by a large margin.
Revised: First, their poor performance on BLI and NER tasks shows that unsupervised joint training fails to generate high-quality alignments due to the lack of a ﬁne-grained seed dictionary as discussed in its limitation 2.

Original: We further remove test pairs of the same surface form (e.g. (hate, hate) as a test pair for en-de) of the BLI task and report their results in Table 2.
Revised: To evaluate accuracy on words that are not shared, we further remove test pairs of the same surface form (e.g. (hate, hate) as a test pair for en-de) of the BLI task and report their results in Table 2.

Original: We ﬁnd unsupervised joint training  to achieve extremely low scores. This is consistent with the PCA visualization shown in Figure 1, where embeddings of non-sharing parts are poorly aligned.
Revised: We ﬁnd unsupervised joint training (row 2) to achieve extremely low scores which shows that emebddings of non-shared parts are poorly aligned, consistent with the PCA visualization shown in Figure 1.

Original: Moreover, we delve into the relative performance of the two paradigms on the MT task by plotting their test BLEU scores of the ﬁrst 20 epochs in Figure 2(a) and 2(b).
Revised: Moreover, we delve into the relative performance of the two paradigms on the MT task by plotting their test BLEU scores of the ﬁrst 20 epochs in Figure 2(a) and 2(b).

Original: We observe that the alignment method actually obtains higher BLEU scores in the ﬁrst few  epochs.
Revised: We observe that the alignment method actually obtains higher BLEU scores in the ﬁrst few epochs, but gets surpassed by joint training in later epochs.

Original: These results verify unsupervised joint training alone cannot align embeddings well. In addition, we can observe from our experiments in MT task that while alignment method performs better in the ﬁrst few epochs, it gets surpassed by joint training in later epochs. This shows the importance of parameter sharing as discussed in limitation 1 of alignment methods. In particular, shared words can be used as a cross-lingual constraint for unsupervised joint training to achieve better ﬁnetuned performance.
Revised: This shows the importance of parameter sharing as discussed in limitationof alignment methods: shared words can be used as a cross-lingual constraint for unsupervised joint training during ﬁne-tuning but this constraint cannot easily be used in alignment methods.

Original: The lack of sharing is also a limitation for  supervised joint training method, which performs poorly on the MT task even with supervision as shown in Figure 2(c).
Revised: The lack of sharing is also a limitation for the supervised joint training method, which performs poorly on the MT task even with supervision as shown in Figure 2(c).

Original: Lastly, we demonstrate that oversharing can be sub-optimal for unsupervised joint training as discussed in its limitation 2.
Revised: Lastly, we demonstrate that oversharing can be sub-optimal for unsupervised joint training as discussed in its limitation 2.

Original: Speciﬁcally, we conduct ablation studies for our framework in Table 1.
Revised: Speciﬁcally, we conduct ablation studies for our framework in Table 1.

Original: Applying alignment reﬁnement on unsupervised joint training without any vocabulary reallocation does not improve its performance.
Revised: Applying alignment reﬁnement on unsupervised joint training without any vocabulary reallocation does not improve its performance (row 13).

Original: On the other hand,a simple vocabulary reallocation alone boosts the performance by quite a margin.
Revised: On the other hand, simple vocabulary reallocation alone boosts the performance by quite a margin (row 11).

Original: This shows some words are shared erroneously across languages in unsupervised joint training thereby hindering its performance.
Revised: This shows some words are shared erroneously across languages in unsupervised joint training, thereby hindering its performance.

Original: Our proposed framework substantially improves over its alignment and joint training baselines on all three tasks. In particular, it outperforms existing methods on all language pairs for the BLI task (using the CSLS as retrieval metric) and achieves state-of-the-art results on 2 out of 3 language pairs for the NER task. Besides, we show that it alleviates limitations of alignment and joint training methods shown in the previous section.
Revised: In addition, while utilizing a seed dictionary (row 10) improves the performance of unsupervised joint train-

Original: [Table]
Revised: [Table]

Original:  that are not directly comparable  denotes supervised XLM model trained with MLM and TLM objectives.
Revised: ‡ denotes results that are not directly comparable due to different resources and architectures ∗ denotes supervised XLM model trained with MLM and TLM objectives.

Original: Its Dutch (nl) result is blank because the model is not pretrained on it.
Revised: Its Dutch (nl) result is blank because the model is not pretrained on it.

Original: Bold signiﬁes state-of-the-art results.
Revised: Bold signiﬁes state-of-the-art results.

Original: We report the average of 5 runs.
Revised: We report the average of 5 runs.

Revised: ing, it still suffers from the oversharing problem and performs worse compared to supervised joint training (row 9).

Revised: 4.3. E VALUATION OF P ROPOSED F RAMEWORK

Revised: As shown in Table 1, Table 3, and Figure 2, our proposed framework substantially improves over the alignment and joint training baselines on all three tasks.

Revised: In particular, it outperforms existing methods on all language pairs for the BLI task (using the CSLS as retrieval metric) and achieves state-of-the-art results on 2 out of 3 language pairs for the NER task.

Revised: Besides, we show that it alleviates limitations of alignment and joint training methods shown in the previous section.

Original: First, the proposed framework largely improves the coarse alignment of unsupervised joint training.
Revised: First, the proposed framework largely improves the poor alignment of unsupervised joint training, especially for non-sharing parts.

Original: As shown in Table 1, the proposed Joint Align framework achieves comparable results to prior methods in the unsupervised case  and it outperforms previous state-of-the-art methods in the supervised setting.
Revised: As shown in Table 1, the proposed Joint Align framework achieves comparable results to prior methods in the unsupervised case (row 12) and it outperforms previous state-of-the-art methods in the supervised setting (row 15).

Original: Speciﬁcally, our proposed framework can generate well-aligned embeddings afteran alignment reﬁnement is applied to the initially ill-aligned embeddings, as demonstrated in Figure 1.
Revised: Speciﬁcally, our proposed framework can generate well-aligned embeddings after alignment reﬁnement is applied to the initially ill-aligned embeddings, as demonstrated in Figure 1.

Original: This is further veriﬁed by results in Table 2, where our proposed framework largely improves accuracy on words not shared between two languages over the unsupervised joint training baseline.
Revised: This is further veriﬁed by results in Table 2, where our proposed framework largely improves accuracy on words not shared between two languages over the unsupervised joint training baseline (row 3 and 6 vs row 2).

Original: Besides, our ablation study in Table 1 further shows the effectiveness of the proposed vocabulary reallocation technique, which alleviates the issue of oversharing.
Revised: Besides, our ablation study in Table 1 further shows the effectiveness of the proposed vocabulary reallocation technique, which alleviates the issue of oversharing.

Original: Particularly, we observe no improvement compared to unsupervised joint training baseline  when an alignment reﬁnement step is used without vocabulary reallocation, while a vocabulary reallocation step alone signiﬁcantly boosts the performance.
Revised: Particularly, we observe no improvement compared to unsupervised joint training baseline (row 8) when an alignment reﬁnement step is used without vocabulary reallocation (row 13), while a vocabulary reallocation step alone signiﬁcantly improves the performance (row 11).

Original: This is consistent with Figure 1 and shows that the oversharing is a bottleneck for applying alignment methods to joint training.
Revised: This is consistent with Figure 1 and shows that the oversharing is a bottleneck for applying alignment methods to joint training.

Original: It also suggests detecting what to share is crucial to achieve better cross-lingual transfer.
Revised: It also suggests detecting what to share is crucial to achieve better cross-lingual transfer.

Original: Lastly, while supervised joint training share the limitation 1 of alignment methods and perform poorly when ﬁnetuned, our proposed frameworktake advantage from unsupervised joint training component and exploits the  idea of word sharing.
Revised: Lastly, while supervised joint training shares the limitation 1 of alignment methods and performs poorly when ﬁnetuned, our proposed framework exploits the same idea of vocabulary sharing used in unsupervised joint training.

Original: In the MT tasks, our framework obtains a maximum gain of 2.97 BLEU over baselines we ran and consistently performs better than results reported in Lample et al.
Revised: In the MT tasks, our framework obtains a maximum gain of 2.97 BLEU over baselines we ran and consistently performs better than results reported in Lample et al.

Original: (2018b).
Revised: (2018b).

Original: In addition, Figure 2 shows that Joint Align not only converges faster in earlier training epochs but also consistently outperforms the two baselines thereafter.
Revised: In addition, Figure 2 shows that Joint Align not only converges faster in earlier training epochs but also consistently outperforms the two baselines thereafter.

Original: These empirical ﬁndings demonstrate the effectiveness of our proposed methods in  non-contextualized case.
Revised: These empirical ﬁndings demonstrate the effectiveness of our proposed methods in the non-contextualized case.

Original: 4.3. C ONTEXTUALIZED W ORD R EPRESENTATIONS
Revised: 4.3. C ONTEXTUALIZED W ORD R EPRESENTATIONS

Original: As can be seen in Table 3, when using our framework, we achieve state-of-the-art results on crosslingual NER on 2 out of 3 languages and the overall average.
Revised: As can be seen in Table 3, when using our framework (M-BERT Feature + Align), we achieve state-of-the-art results on cross-lingual NER on 2 out of 3 languages and the overall average.

Original: It shows that our framework can effectively generalize to contextualized representations.
Revised: This shows that our framework can effectively generalize to contextualized representations.

Original: Speciﬁcally, our framework improves over  the M-BERT feature extraction baseline on all three language pairs and outperforms the M-BERT ﬁnetuning counterparts.
Revised: Speciﬁcally, our framework improves over both the M-BERT feature extraction and ﬁnetuning baselines on all three language pairs. However, when compared to non-contextualized results, the gain of using alignment reﬁnement on top of unsupervised joint training is much smaller. This suggests that, as the contextualized unsupervised joint training model performs very well already even without [Figure]

Original: The reason why acontextualized supervised joint training model, XLM, performs worse than its unsupervised counterpart, M-BERT, is likely that XLM uses an uncased vocabulary, where casing information is important for NER tasks.
Revised: any supervision, it is harder to achieve large improvements. While alignment reﬁnement relies on word alignment, a process that is noisy itself, a better alignment approach may be warranted. Lastly, the reason why a supervised joint training model, XLM, performs worse than its unsupervised counterpart, M-BERT, is likely that XLM uses an uncased vocabulary, where casing information is important for NER tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Word embeddings (Mikolov et al., 2013a) are a key ingredient to achieve success in monolingual NLP tasks.
Revised: Word embeddings (Mikolov et al., 2013a; Ruder et al., 2019) are a key ingredient to achieving success in monolingual NLP tasks.

Original: However, directly using word embeddings independently trained for each language may cause negative transfer (Wang et al., 2019) in cross-lingual transfer tasks.
Revised: However, directly using word embeddings independently trained for each language may cause negative transfer (Wang et al., 2019) in cross-lingual transfer tasks.

Original: In order to capture the cross-lingual mapping, a rich body of existing works relying on cross-lingual supervisions, including bilingual dictionaries (Mikolov et al., 2013a; Faruqui & Dyer, 2014; Artetxe et al., 2016; Xing et al., 2015; Duong et al., 2016; Gouws & Søgaard, 2015; Joulin et al., 2018a), sentence-aligned corpora (Koˇcisk`y et al., 2014; Hermann & Blunsom, 2014; Gouws et al., 2015) and documentaligned corpora (Vuli´c & Moens, 2016; Søgaard et al., 2015).[Figure]
Revised: In order to capture the cross-lingual mapping, a rich body of existing work relying on cross-lingual supervision, including bilingual dictionaries (Mikolov et al., 2013a; Faruqui & Dyer, 2014; Artetxe et al., 2016; Xing et al., 2015; Duong et al., 2016; Gouws & Søgaard, 2015; Joulin et al., 2018a), sentence-aligned corpora (Ko ˇ cisk ` y et al., 2014; Hermann & Blunsom, 2014; Gouws et al., 2015) and document-aligned corpora (Vuli ´ c & Moens, 2016; Søgaard et al., 2015).

Original: Besides, unsupervised alignment methods aim to eliminate the requirement for cross-lingual supervision.
Revised: Besides, unsupervised alignment methods aim to eliminate the requirement for cross-lingual supervision.

Original: Early work of Cao et al.
Revised: Early work of Cao et al.

Original: (2016) matches the mean and the standard deviation of two embedding spaces after alignment.
Revised: (2016) matches the mean and the standard deviation of two embedding spaces after alignment.

Original: Barone (2016); Zhang et al.
Revised: Barone (2016); Zhang et al.

Original: (2017a;b); Conneau et al.
Revised: (2017a;b); Conneau et al.

Original: (2018a) adapted a generative adversarial network (GAN) (Goodfellow et al., 2014) to make the distributions of two word embedding spaces indistinguishable.
Revised: (2018a) adapted a generative adversarial network (GAN) (Goodfellow et al., 2014) to make the distributions of two word embedding spaces indistinguishable.

Original: Follow-up works improve uponthe GAN-based training for better stability and robustness by introducing Sinkhorn distance (Xu et al., 2018), by stochastic self-training (Artetxe et al., 2018a), or by introducing latent variables (Dou et al., 2018).
Revised: Follow-up works improve upon GAN-based training for better stability and robustness by introducing Sinkhorn distance (Xu et al., 2018), by stochastic self-training (Artetxe et al., 2018a), or by introducing latent variables (Dou et al., 2018).

Original: While alignment methods utilize embeddings trained independently on different languages, joint training methods train word embeddings at the same time.
Revised: While alignment methods utilize embeddings trained independently on different languages, joint training methods train word embeddings at the same time.

Original: Klementiev et al.
Revised: Klementiev et al.

Original: (2012b) train a bilingual dictionary-based regularization term jointly with monolingual language model objectives while
Revised: (2012) train a bilingual dictionary-based regularization term jointly with monolingual language model objectives while

Original: Koˇcisk`y et al.
Revised: Koˇcisk`y et al.

Original: (2014) deﬁnes the cross-lingual regularization with the parallel corpus.
Revised: (2014) deﬁnes the cross-lingual regularization with the parallel corpus.

Original: Another branch of methods (Xiao & Guo, 2014; Gouws & Søgaard, 2015; Ammar et al., 2016; Duong et al., 2016) build a pseudo-bilingual corpus by randomly replacing words in monolingual corpus with their translations and use monolingual word embedding algorithms to induce bilingual representations.
Revised: Another branch of methods (Xiao & Guo, 2014; Gouws & Søgaard, 2015; Ammar et al., 2016; Duong et al., 2016) build a pseudo-bilingual corpus by randomly replacing words in monolingual corpus with their translations and use monolingual word embedding algorithms to induce bilingual representations.

Original: The unsupervised joint method by Lample & Conneau (2019) simply exploit words that share the same surface form as bilingual “supervision” and directly train a shared set of embedding with joint vocabulary.
Revised: The unsupervised joint method by Lample & Conneau (2019) simply exploit words that share the same surface form as bilingual “supervision” and directly train a shared set of embedding with joint vocabulary.

Original: Recently, unsupervised joint training of contextualized word embeddings through the form of multilingual language model pretraining using shared subword vocabularies has produced state-of-the-art results on various benchmarks(Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019; Pires et al., 2019; Wu & Dredze, 2019).
Revised: Recently, unsupervised joint training of contextualized word embeddings through the form of multilingual language model pretraining using shared subword vocabularies has produced state-of-the-art results on various benchmarks

Original: A concurrent work by Ormazabal et al.
Revised: A concurrent work by Ormazabal et al.

Original: (2019) also compares alignment and joint method in the bilingual lexicon induction task.
Revised: (2019) also compares alignment and joint method in the bilingual lexicon induction task.

Original: Different from their setup which only tests on supervised settings, we conduct analysis across various tasks and experiment with both supervised and unsupervised conditions.
Revised: Different from their setup which only tests on supervised settings, we conduct analysis across various tasks and experiment with both supervised and unsupervised conditions.

Original: While Ormazabal et al.
Revised: While Ormazabal et al.

Original: (2019) suggests  the combination of the alignment and joint model could potentially advance the state-of-art of both worlds, we propose  anovel training framework and empirically veriﬁed its effectiveness on various tasks and settings.
Revised: (2019) suggests that the combination of the alignment and joint model could potentially advance the state-of-art of both worlds, we propose such a framework and empirically verify its effectiveness on various tasks and settings.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we systematically compare the alignment and joint training methods for CLWE.
Revised: In this paper, we systematically compare the alignment and joint training methods for CLWE.

Original: We point out that the nature of each category of methods leads to certain strengths and limitations.
Revised: We point out that the nature of each category of methods leads to certain strengths and limitations.

Original: The empirical experiments on extensive benchmark datasets and various NLP tasks veriﬁed our analysis.
Revised: The empirical experiments on extensive benchmark datasets and various NLP tasks veriﬁed our analysis.

Original: To further improve the state-of-art of CLWE, we propose a simple hybrid framework which combines the strength from both worlds and achieves signiﬁcantly better performance in the
Revised: To further improve the state-of-art of CLWE, we propose a simple hybrid framework which combines the strength from both worlds and achieves signiﬁcantly better performance in the

Original: BLI, MT and NER tasks.
Revised: BLI, MT and NER tasks.

Original: Our work opens a promising new direction that combines two previously exclusive lines of research.
Revised: Our work opens a promising new direction that combines two previously exclusive lines of research.

Original: For future work, an interesting direction is to ﬁnd a more optimal word sharing strategy. 
Revised: For future work, an interesting direction is to ﬁnd a more effective word sharing strategy. Acknowledgments:

Revised: This research was sponsored by Defense Advanced Research Projects Agency

Revised: Information Innovation Ofﬁce (I2O) under the Low Resource Languages for Emergent Incidents (LORELEI) program, issued by DARPA/I2O under Contract No. HR0011-15-C0114.

Revised: The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the U.S. government.

Revised: The U.S. government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation here on.

Revised: APPENDIX A NER E XPERIMENT D ETAILS

Revised: Here we include some additional details for the NER experiments with contextualized representations:

Revised: • Alignment As described in Section 3.2, we apply word alignment methods, such as fastalign (Dyer et al., 2013), on parallel data to extract word-aligned pairs for learning the alignment matrix.

Revised: As M-BERT is based on subword tokens, we use the average of the representations of all subword tokens that correspond to a word as the representation for that word.

Revised: For instance, assume that an English word “Resumption” is aligned to a German word “Wiederaufnahme”, and they are tokenized by M-BERT as “Res”, “##sumption”, and “Wie”, “##dera”, “##uf”, “##nahme”, respectively.

Revised: Then the representation for “Resumption” is the average of the representations of subword tokens “Res” and “##sumption”, and the same goes for “Wiederaufnahme”.

Revised: • Hyperparameters For the task-speciﬁc NER model, we use a 2-layer Bi-LSTM with a hidden size of 768 followed by a CRF layer.

Revised: We apply a dropout rate of 0.5 on the input

Revised: en-es es-en en-fr fr-en en-de de-en en-it it-en en-ru ru-en en-zh

Revised: zh-

Revised: [Table]

Revised: AlignmentMethods

Revised: [Table]

Revised: (Conneauetal.,2018a)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: (Smithetal.,2017)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Jawanpuriaetal

Revised: [Table]

Revised: [Table]

Revised: (Joulinetal.,2018b)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 42.6 (10)Joint Align+MUSE 82.

Revised: 68.7 (11)Joint Align+RCSLS(w/oVR) 34.

Revised: 29.2 (12)Joint Align+GeoMM 83.

Revised: 73.3 (13)Joint Align+RCSLS 87.

Revised: and the output of the Bi-LSTM, and use Adam with default parameters and a learning rate of 0.0001 for optimization.

Revised: We train the model for 40 epochs with a batch size of 10, and evaluate the model per 150 steps.

Revised: For prediction, we feed the outputs of the Bi-LSTM that correspond to the ﬁrst subword tokens of each word to the CRF model.

Revised: This is identical to ﬁnetuning BERT on the NER task, except that in our case the outputs that correspond to the ﬁrst subword token are fed into a CRF, rather than a linear layer as done in BERT.

Revised: B BLI T EST P AIRS

Revised: The ofﬁcial evaluation script 8 of MUSE only includes test pairs whose source words and target words both appear in their corresponding vocabularies, leaving out those that are OOV on either side.

Revised: Since our proposed vocabulary reallocation step modiﬁes both the source and target vocabularies, the script may exclude some test pairs when evaluating our model.

Revised: For example, the word “age” from the test pair (age, age) for en-fr could be allocated as an en (not shared ) word, so it is OOV on the fr side and the pair would thus be left out by the script.

Revised: As a result, the total number of test pairs would be smaller, thereby changing the denominator when we calculate accuracy.

Revised: To ensure fair comparison, we include these OOV pairs so the total number of test pairs stays the same.

Revised: Speciﬁcally, if the source word of a test pair is OOV, we retrieve itself.

Revised: Otherwise, we count it as incorrect.

Revised: In addition, we further investigate which pairs are left out and found that the MUSE benchmark contains some noisy test data.

Revised: Speciﬁcally, we ﬁnd that the majority of these pairs are in the same surface form, such as (sit, sit), but many target words are not actual translations of the source words.

Revised: For example, we found test pairs such as { (age, age), (century, century) } for en-fr and { (mickey, mickey), (uncredited, uncredited) } for en-zh .

Revised: Clearly, these are English words and should not be considered as appropriate translations for French or Chinese.

Revised: In Table 1, we mark these pairs as incorrect for our framework to ensure fair comparison, while in fact our framework correctly allocates these words to English.

Revised: To reveal the full picture, we also conduct BLI experiments without test pairs that got left out due to vocabulary reallocation.

Revised: The results are shown in Table 4.

Revised: We observe that our proposed framework obtains a gain of 3.9 accuracy on average over the RCSLS baseline.