Original: F ILTERED I NNER P RODUCT P ROJECTION FOR C ROSSLINGUAL E MBEDDING A LIGNMENT
Revised: F ILTERED I NNER P RODUCT P ROJECTION FOR C ROSSLINGUAL E MBEDDING A LIGNMENT

Original: A BSTRACT
Revised: A BSTRACT

Original: Due to widespread interest in machine translation and transfer learning, there are numerous algorithms for mapping multiple embeddings to a shared representation space.
Revised: Due to widespread interest in machine translation and transfer learning, there are numerous algorithms for mapping multiple embeddings to a shared representation space.

Original: Recently, these algorithms have been studied in the setting of bilingual lexicon induction where one seeks to align the embeddings of a source and a target language such that translated word pairs lie close to one another in a common representation space.
Revised: Recently, these algorithms have been studied in the setting of bilingual lexicon induction where one seeks to align the embeddings of a source and a target language such that translated word pairs lie close to one another in a common representation space.

Original: In this paper, we propose a method, Filtered Inner Product
Revised: In this paper, we propose a method, Filtered Inner Product

Original: Projection (FIPP), for mapping embeddings to a common representation space.
Revised: Projection (FIPP), for mapping embeddings to a common representation space.

Original: As semantic shifts are pervasive across languages and domains, FIPP ﬁrst identiﬁes the common geometric structure in both embeddings and then, only on the common structure, aligns the Gram matrices of these embeddings.
Revised: As semantic shifts are pervasive across languages and domains, FIPP ﬁrst identiﬁes the common geometric structure in both embeddings and then, only on the common structure, aligns the Gram matrices of these embeddings.

Original: FIPP aligns embeddings to isomorphic vector spaces even when the source and target embeddings are of differing dimensionalities.
Revised: FIPP aligns embeddings to isomorphic vector spaces even when the source and target embeddings are of differing dimensionalities.

Original: Additionally, FIPP provides computational beneﬁts in ease of implementation and is faster to compute than current approaches.
Revised: Additionally, FIPP provides computational beneﬁts in ease of implementation and is faster to compute than current approaches.

Original: Following the baselines in Glavaš et al.
Revised: Following the baselines in Glavaš et al.

Original: (2019), we evaluate FIPP in the context of bilingual lexicon induction and downstream language tasks.
Revised: (2019), we evaluate FIPP in the context of bilingual lexicon induction and downstream language tasks.

Original: We show that FIPP outperforms existing methods on the XLING (5K) BLI dataset and the XLING (1K) BLI dataset, when using a self-learning approach, while also providing robust performance across downstream tasks.
Revised: We show that FIPP outperforms existing methods on the XLING (5K) BLI dataset and the XLING (1K) BLI dataset, when using a self-learning approach, while also providing robust performance across downstream tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The problem of aligning sets of embeddings, or high dimensional real valued vectors, is of great interest in natural language processing, with applications in machine translation and transfer learning, and shares connections to graph matching and assignment problems (Grave et al., 2019; Gold &
Revised: The problem of aligning sets of embeddings, or high dimensional real valued vectors, is of great interest in natural language processing, with applications in machine translation and transfer learning, and shares connections to graph matching and assignment problems (Grave et al., 2019; Gold &

Original: Rangarajan, 1996).
Revised: Rangarajan, 1996).

Original: Aligning embeddings trained on corpora from different languages has led to improved performance of supervised and unsupervised word and sentence translation (Zou et al., 2013), sequence labeling (Zhang et al., 2016;
Revised: Aligning embeddings trained on corpora from different languages has led to improved performance of supervised and unsupervised word and sentence translation (Zou et al., 2013), sequence labeling (Zhang et al., 2016;

Original: Mayhew et al., 2017), and information retrieval (Vuli´c & Moens, 2015).
Revised: Mayhew et al., 2017), and information retrieval (Vuli´c & Moens, 2015).

Original: Additionally, linguistic patterns have been studied using embedding alignment algorithms (Schlechtweg et al., 2019; Lauscher & Glavaš, 2019).
Revised: Additionally, linguistic patterns have been studied using embedding alignment algorithms (Schlechtweg et al., 2019; Lauscher & Glavaš, 2019).

Original: Embedding alignments have also been shown to improve the performance of multilingual contextual representation models (i.e. mBERT), when used during intialization, on certain tasks such as multilingual document classiﬁcation (Artetxe et al., 2020)
Revised: Embedding alignments have also been shown to improve the performance of multilingual contextual representation models (i.e. mBERT), when used during intialization, on certain tasks such as multilingual document classiﬁcation (Artetxe et al., 2020)

Original: Recently, algorithms using embedding alignments on the input token representations of contextual embedding models have been shown to provide efﬁcient domain adaptation (Poerner et al., 2020).
Revised: Recently, algorithms using embedding alignments on the input token representations of contextual embedding models have been shown to provide efﬁcient domain adaptation (Poerner et al., 2020).

Original: Lastly, aligned source and target input embeddings have been shown to improve the transferability of models learned on a source domain to a target domain (Artetxe et al., 2018a; Wang et al., 2018; Mogadala & Rettinger, 2016).
Revised: Lastly, aligned source and target input embeddings have been shown to improve the transferability of models learned on a source domain to a target domain (Artetxe et al., 2018a; Wang et al., 2018; Mogadala & Rettinger, 2016).

Original: In the bilingual lexicon induction task, one seeks to learn a transformation on the embeddings of a source and a target language so that translated word pairs lie close to one another in the shared representation space.
Revised: In the bilingual lexicon induction task, one seeks to learn a transformation on the embeddings of a source and a target language so that translated word pairs lie close to one another in the shared representation space.

Original: Speciﬁcally, one is given a small seed dictionary D containing c pairs of translated words, and embeddings for these word pairs in a source and a target language, X s ∈ R c × d and X t ∈ R c × d .
Revised: Speciﬁcally, one is given a small seed dictionary D containing c pairs of translated words, and embeddings for these word pairs in a source and a target language, X s ∈ R c × d and X t ∈ R c × d .

Original: Using this seed dictionary, a transformation is learned on X s and X t with the objective that unseen translation pairs can be induced, often through nearest neighbors search.
Revised: Using this seed dictionary, a transformation is learned on X s and X t with the objective that unseen translation pairs can be induced, often through nearest neighbors search.

Original: Previous literature on this topic has focused on aligning embeddings by minimizing matrix or distributional distances (Grave et al., 2019; Jawanpuria et al., 2019;
Revised: Previous literature on this topic has focused on aligning embeddings by minimizing matrix or distributional distances (Grave et al., 2019; Jawanpuria et al., 2019;

Original: Joulin et al., 2018a).
Revised: Joulin et al., 2018a).

Original: For instance,
Revised: For instance,

Original: Mikolov et al.
Revised: Mikolov et al.

Original: (2013a) proposed using Stochastic Gradient Descent (SGD) to learn a mapping, Ω , to minimize the sum of squared distances between pairs of words in the seed dictionary (cid:107) X Ds Ω
Revised: (2013a) proposed using Stochastic Gradient Descent (SGD) to learn a mapping, Ω , to minimize the sum of squared distances between pairs of words in the seed dictionary (cid:107) X Ds Ω

Original: − X Dt (cid:107) 2 F , which achieves high word translation accuracy for similar languages.
Revised: − X Dt (cid:107) 2 F , which achieves high word translation accuracy for similar languages.

Original: Smith et al.
Revised: Smith et al.

Original: (2017) and Artetxe et al.
Revised: (2017) and Artetxe et al.

Original: (2017) independently showed that a mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the
Revised: (2017) independently showed that a mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the

Original: Orthogonal Procrustes problem, Ω ∗ =
Revised: Orthogonal Procrustes problem, Ω ∗ =

Original: arg min Ω ∈ O ( d ) (cid:107) X Ds Ω − X Dt (cid:107) F where O ( d ) the the group of d dimensional orthogonal matrices.
Revised: arg min Ω ∈ O ( d ) (cid:107) X Ds Ω − X Dt (cid:107) F where O ( d ) the the group of d dimensional orthogonal matrices.

Original: However, these methods usually require the dimensions of the source and target language embeddings to be the same, which often may not hold.
Revised: However, these methods usually require the dimensions of the source and target language embeddings to be the same, which often may not hold.

Original: Furthermore, due to semantic shifts across languages, it’s often the case that a word and its translation may not co-occur with the same sets of words (Gulordava & Baroni, 2011).
Revised: Furthermore, due to semantic shifts across languages, it’s often the case that a word and its translation may not co-occur with the same sets of words (Gulordava & Baroni, 2011).

Original: Therefore, seeking an alignment which minimizes all pairwise distances among translated pairs results in using information not common to both the source and target embeddings.
Revised: Therefore, seeking an alignment which minimizes all pairwise distances among translated pairs results in using information not common to both the source and target embeddings.

Original: To address these problems, we propose Filtered Inner Product Projection (FIPP) for mapping embeddings from different languages to a shared representation space.
Revised: To address these problems, we propose Filtered Inner Product Projection (FIPP) for mapping embeddings from different languages to a shared representation space.

Original: FIPP aligns a source embedding
Revised: FIPP aligns a source embedding

Original: X s ∈ R n × d 1 to a target embedding X t ∈ R m × d 2 and maps vectors in X s to the R d 2 space of X t .
Revised: X s ∈ R n × d 1 to a target embedding X t ∈ R m × d 2 and maps vectors in X s to the R d 2 space of X t .

Original: Instead of word-level information, FIPP focuses on pairwise distance information, speciﬁed by the
Revised: Instead of word-level information, FIPP focuses on pairwise distance information, speciﬁed by the

Original: Gram matrices
Revised: Gram matrices

Original: X s X Ts and X t X Tt , where the rows of X s and X t correspond to embeddings for the c pairs of source and target words from the seed dictionary.
Revised: X s X Ts and X t X Tt , where the rows of X s and X t correspond to embeddings for the c pairs of source and target words from the seed dictionary.

Original: During alignment, FIPP tries to achieve the following two goals.
Revised: During alignment, FIPP tries to achieve the following two goals.

Original: First, it is desired that the aligned source embedding FIPP ( X s ) =
Revised: First, it is desired that the aligned source embedding FIPP ( X s ) =

Original: [Equation]
Revised: [Equation]

Original: be structurally close to the original source embedding to ensure that semantic information is retained and prevent against overﬁtting on the seed dictionary.
Revised: be structurally close to the original source embedding to ensure that semantic information is retained and prevent against overﬁtting on the seed dictionary.

Original: This goal is reﬂected in the minimization of the reconstruction loss :
Revised: This goal is reﬂected in the minimization of the reconstruction loss :

Original: [Equation]
Revised: [Equation]

Original: Second, as the usage of words and their translations vary across languages, instead of requiring ˜ X s to use all of the distance information from X t , FIPP selects a ﬁltered set K of word pairs that have similar distances in both the source and target languages:
Revised: Second, as the usage of words and their translations vary across languages, instead of requiring ˜ X s to use all of the distance information from X t , FIPP selects a ﬁltered set K of word pairs that have similar distances in both the source and target languages:

Original: K = { (
Revised: K = { (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: FIPP then minimizes a transfer loss on this set K , the squared difference in distances between the aligned source embeddings and the target embeddings: (cid:80)
Revised: FIPP then minimizes a transfer loss on this set K , the squared difference in distances between the aligned source embeddings and the target embeddings: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: from X t and mapped to R d 

Revised: from X t and mapped to R d

Original: while X t
Revised: while X t

Original: is
Revised: is

Original: unchanged
Revised: unchanged

Original: transformations as FIPP’s alignment stems directly from inner product information.
Revised: transformations as FIPP’s alignment stems directly from inner product information.

Original: There are numerous beneﬁts to aligning embeddings based on inner products in terms of applicability, interpretability, and efﬁciency.
Revised: There are numerous beneﬁts to aligning embeddings based on inner products in terms of applicability, interpretability, and efﬁciency.

Original: First, since FIPP’s alignment between the source and target embeddings is performed on the inner product matrices, X s X Ts and X t X
Revised: First, since FIPP’s alignment between the source and target embeddings is performed on the inner product matrices, X s X Ts and X t X

Original: Tt 2 R n ⇥ n , embeddings are not required to be of the same dimensions.
Revised: Tt 2 R n ⇥ n , embeddings are not required to be of the same dimensions.

Original: This is particularly helpful for transferring relevant knowledge from embeddings trained on large corpuses to smaller/domain-speciﬁc embeddings such as in the case of machine translation for low resource languages.
Revised: This is particularly helpful for transferring relevant knowledge from embeddings trained on large corpuses to smaller/domain-speciﬁc embeddings such as in the case of machine translation for low resource languages.

Original: Additionally, dimensionality-invariant alignment can be helpful in compressing input representa- tions for memory and computation sensitive settings.
Revised: Additionally, dimensionality-invariant alignment can be helpful in compressing input representa- tions for memory and computation sensitive settings.

Original: Secondly, alignment modiﬁcations on pairwise inner prod- ucts provide more granularity and increased interpretability in comparison to dimension based approaches.
Revised: Secondly, alignment modiﬁcations on pairwise inner prod- ucts provide more granularity and increased interpretability in comparison to dimension based approaches.

Original: This is be-
Revised: This is be-

Original: cause FIPP’s conversion of source embeddings is explicitly linked to speciﬁc changes in inner products between words.
Revised: cause FIPP’s conversion of source embeddings is explicitly linked to speciﬁc changes in inner products between words.

Original: Additionally, FIPP allows better control over alignment as it’s transfer loss can be altered to only modify angles be-
Revised: Additionally, FIPP allows better control over alignment as it’s transfer loss can be altered to only modify angles be-

Original: tween speciﬁed word pairs.
Revised: tween speciﬁed word pairs.

Original: Lastly, since alignment can be parallelized on a word-pair basis we ﬁnd that FIPP provides improvements in computational efﬁciency.
Revised: Lastly, since alignment can be parallelized on a word-pair basis we ﬁnd that FIPP provides improvements in computational efﬁciency.

Original: We conduct a thorough evaluation of FIPP for multilingual embedding alignment on the MUSE word translation task set.
Revised: We conduct a thorough evaluation of FIPP for multilingual embedding alignment on the MUSE word translation task set.

Original: In comparison to existing approaches, we ﬁnd that FIPP is more computationally efﬁcient, can be applied to more general settings, and achieves better word translation performance for dissimilar language pairs.
Revised: In comparison to existing approaches, we ﬁnd that FIPP is more computationally efﬁcient, can be applied to more general settings, and achieves better word translation performance for dissimilar language pairs.

Original: We also discuss how FIPP can be directly applied in other applications such as embedding compression and transfer learning in natural language processing.
Revised: We also discuss how FIPP can be directly applied in other applications such as embedding compression and transfer learning in natural language processing.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: We discuss related work in Section 2.
Revised: We discuss related work in Section 2.

Original: We introduce our FIPP model in Section 3.
Revised: We introduce our FIPP model in Section 3.

Original: We present experimental results in Sectionand further discuss ﬁndings in Section 5.
Revised: We present experimental results in Sectionand further discuss ﬁndings in Section 5.

Original: Lastly, we discuss related applications of FIPP in Section 6 and conclude our paper in Section 7.
Revised: Lastly, we discuss related applications of FIPP in Section 6 and conclude our paper in Section 7.

Original: 2. Related Work
Revised: 2. Related Work

Original: In this section, we discuss related work in quantifying semantic shifts in embeddings and the task of multilingual embedding alignment.
Revised: In this section, we discuss related work in quantifying semantic shifts in embeddings and the task of multilingual embedding alignment.

Original: 2.1. Distributional Methods for Quantifying Semantic Shifts
Revised: 2.1. Distributional Methods for Quantifying Semantic Shifts

Original: Prior work has shown monolingual text corpora from different communities or time periods exhibit variations in semantics and syntax (Hamilton et al., 2016a;b).
Revised: Prior work has shown monolingual text corpora from different communities or time periods exhibit variations in semantics and syntax (Hamilton et al., 2016a;b).

Original: In order to ﬁnd linguistic shifts in different corpora, distributional methods make comparisons on word co-occurence distributions.
Revised: In order to ﬁnd linguistic shifts in different corpora, distributional methods make comparisons on word co-occurence distributions.

Original: These approaches have been well studied to classify the introduction of new words and senses (Jatowt & Duh, 2014).
Revised: These approaches have been well studied to classify the introduction of new words and senses (Jatowt & Duh, 2014).

Original: Word embeddings (Mikolov et al., 2013b; Pennington et al., 2014;
Revised: Word embeddings (Mikolov et al., 2013b; Pennington et al., 2014;

Original: Bojanowski et al., 2017) map words to representations in a continuous space.
Revised: Bojanowski et al., 2017) map words to representations in a continuous space.

Original: By comparing pairwise distances in embeddings trained on separate corpora, one can determine semantic shifts associated with biases and cultural norms (Gulordava & Baroni, 2011; Sagi et al.
Revised: By comparing pairwise distances in embeddings trained on separate corpora, one can determine semantic shifts associated with biases and cultural norms (Gulordava & Baroni, 2011; Sagi et al.

Original: ; Kim et al., 2014).
Revised: ; Kim et al., 2014).

Original: [Table]
Revised: [Table]

Original: has been
Revised: has been

Original: [Table]
Revised: [Table]

Original: in many
Revised: in many

Original: [Table]
Revised: [Table]

Original: , e.g. including the
Revised: , e.g. including the

Original: [Table]
Revised: [Table]

Original: Of-
Revised: Of-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: (Zou et al., 2013).
Revised: (Zou et al., 2013).

Original: Recently proposed methods extend this notion by comparing all
Revised: Recently proposed methods extend this notion by comparing all

Original: [Table]
Revised: [Table]

Original: of words and
Revised: of words and

Original: more speciﬁcally the
Revised: more speciﬁcally the

Original: [Table]
Revised: [Table]

Original: (Yin
Revised: (Yin

Original: [Table]
Revised: [Table]

Original: al., 2018).
Revised: al., 2018).

Original: However, these
Revised: However, these

Original: [Table]
Revised: [Table]

Original: language
Revised: language

Original: [Table] [Table]
Revised: [Table] [Table]

Original: have not yet been explored in the
Revised: have not yet been explored in the

Original: [Table]
Revised: [Table]

Original: of
Revised: of

Original: [Table]
Revised: [Table]

Original: or
Revised: or

Original: [Table]
Revised: [Table]

Original: . Word Translation with Multilingual Embedding Alignment
Revised: . Word Translation with Multilingual Embedding Alignment

Original: The problem of word translation from multilingual embed- dings has been studied extensively in natural language pro-
Revised: The problem of word translation from multilingual embed- dings has been studied extensively in natural language pro-

Original: cessing, under both supervised and unsupervised learning settings.
Revised: cessing, under both supervised and unsupervised learning settings.

Original: For unsupervised word translation (Artetxe et al., 2017), one seeks to align word embeddings trained on a source and target embedding, X s and X t respectively, such that for any source word, the closest word embedding in the target embedding space is the source word’s translation.
Revised: For unsupervised word translation (Artetxe et al., 2017), one seeks to align word embeddings trained on a source and target embedding, X s and X t respectively, such that for any source word, the closest word embedding in the target embedding space is the source word’s translation.

Original: The supervised learning setting, commonly known as the
Revised: The supervised learning setting, commonly known as the

Original: We show FIPP can be efﬁciently solved using either low-rank semideﬁnite approximations or with stochastic gradient descent.
Revised: We show FIPP can be efﬁciently solved using either low-rank semideﬁnite approximations or with stochastic gradient descent.

Original: Also, we formulate a least squares projection to infer aligned representations for words outside the seed dictionary and present a weighted Procrustes objective which recovers an orthogonal operator that takes into consideration the degree of structural similarity among translation pairs.
Revised: Also, we formulate a least squares projection to infer aligned representations for words outside the seed dictionary and present a weighted Procrustes objective which recovers an orthogonal operator that takes into consideration the degree of structural similarity among translation pairs.

Original: The method is illustrated in Figure 1.
Revised: The method is illustrated in Figure 1.

Original: Compared to previous approaches, FIPP has improved generality, stability, and efﬁciency.
Revised: Compared to previous approaches, FIPP has improved generality, stability, and efﬁciency.

Original: First, since FIPP’s alignment between the source and target embeddings is performed on Gram matrices, i.e. X s X Ts and X
Revised: First, since FIPP’s alignment between the source and target embeddings is performed on Gram matrices, i.e. X s X Ts and X

Original: t X Tt ∈ R c × c , embeddings are not required to be of the same dimension and are projected to isomorphic vector spaces.
Revised: t X Tt ∈ R c × c , embeddings are not required to be of the same dimension and are projected to isomorphic vector spaces.

Original: This is particularly helpful for aligning embeddings trained on smaller corpora, such as in low resource domains, or compute-intensive settings where embeddings may have been compressed to lower dimensions.
Revised: This is particularly helpful for aligning embeddings trained on smaller corpora, such as in low resource domains, or compute-intensive settings where embeddings may have been compressed to lower dimensions.

Original: Secondly, alignment modiﬁcations made on ﬁltered Gram matrices can incorporate varying constraints on alignment at the most granular level, pairwise distances.
Revised: Secondly, alignment modiﬁcations made on ﬁltered Gram matrices can incorporate varying constraints on alignment at the most granular level, pairwise distances.

Original: Lastly, FIPP is easy to implement as it involves only matrix operations, is deterministic, and takes an order of magnitude less time to compute than either the best supervised (Joulin et al., 2018b) or unsupervised approach (Artetxe et al., 2018c) compared against.
Revised: Lastly, FIPP is easy to implement as it involves only matrix operations, is deterministic, and takes an order of magnitude less time to compute than either the best supervised (Joulin et al., 2018b) or unsupervised approach (Artetxe et al., 2018c) compared against.

Original: We conduct a thorough evaluation of FIPP using baselines outlined in Glavaš et al.
Revised: We conduct a thorough evaluation of FIPP using baselines outlined in Glavaš et al.

Original: (2019) including bilingual lexicon induction with 5 K and 1 K supervision sets and downstream evaluation on MNLI
Revised: (2019) including bilingual lexicon induction with 5 K and 1 K supervision sets and downstream evaluation on MNLI

Original: Natural Language Inference and Ted CLDC Document Classiﬁcation tasks.
Revised: Natural Language Inference and Ted CLDC Document Classiﬁcation tasks.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: We discuss related work in Section 2.
Revised: We discuss related work in Section 2.

Original: We introduce our FIPP model in Section 3 and usage for inference in Section 4.
Revised: We introduce our FIPP model in Section 3 and usage for inference in Section 4.

Original: We present experimental results in Section 5 and further discuss ﬁndings in Section 6.
Revised: We present experimental results in Section 5 and further discuss ﬁndings in Section 6.

Original: We conclude the paper in Section 7.
Revised: We conclude the paper in Section 7.

Original:  R ELATED W ORK 2. D ISTRIBUTIONAL M ETHODS FOR Q UANTIFYING S EMANTIC S HIFTS
Revised:  R ELATED W ORK 2. D ISTRIBUTIONAL M ETHODS FOR Q UANTIFYING S EMANTIC S HIFTS

Original: Prior work has shown that monolingual text corpora from different communities or time periods exhibit variations in semantics and syntax (Hamilton et al., 2016a;b).
Revised: Prior work has shown that monolingual text corpora from different communities or time periods exhibit variations in semantics and syntax (Hamilton et al., 2016a;b).

Original: Word embeddings (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017) map words to representations in a continuous space with the objective that the inner product between any two words representations is approximately proportional to their probability of co-occurrence.
Revised: Word embeddings (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017) map words to representations in a continuous space with the objective that the inner product between any two words representations is approximately proportional to their probability of co-occurrence.

Original: By comparing pairwise distances in monolingual embeddings trained on separate corpora, one can quantify semantic shifts associated with biases, cultural norms, and temporal differences (Gulordava & Baroni, 2011; Sagi et al., 2011;
Revised: By comparing pairwise distances in monolingual embeddings trained on separate corpora, one can quantify semantic shifts associated with biases, cultural norms, and temporal differences (Gulordava & Baroni, 2011; Sagi et al., 2011;

Original: Kim et al., 2014).
Revised: Kim et al., 2014).

Original: Recently proposed metrics on embeddings compare all pairwise inner products of two embeddings, E and F , of the form (cid:107) EE T − FF T (cid:107) F
Revised: Recently proposed metrics on embeddings compare all pairwise inner products of two embeddings, E and F , of the form (cid:107) EE T − FF T (cid:107) F

Original: (Yin et al., 2018).
Revised: (Yin et al., 2018).

Original: While these metrics have been applied in quantifying monolingual semantic variation, they have not been explored in context of mapping embeddings to a common representation space or in multilingual settings.
Revised: While these metrics have been applied in quantifying monolingual semantic variation, they have not been explored in context of mapping embeddings to a common representation space or in multilingual settings.

Original: 2. C ROSSLINGUAL E MBEDDING A LIGNMENT
Revised: 2. C ROSSLINGUAL E MBEDDING A LIGNMENT

Original: The ﬁrst work on this topic is by Mikolov et al.
Revised: The ﬁrst work on this topic is by Mikolov et al.

Original: (2013a) who proposed using Stochastic Gradient
Revised: (2013a) who proposed using Stochastic Gradient

Original: Descent (SGD) to learn a mapping, Ω , to minimize the sum of squared distances between pairs of words in the seed dictionary (cid:107) X
Revised: Descent (SGD) to learn a mapping, Ω , to minimize the sum of squared distances between pairs of words in the seed dictionary (cid:107) X

Original: s Ω − X t (cid:107) 2 F , which achieves high word translation accuracy for similar languages.
Revised: s Ω − X t (cid:107) 2 F , which achieves high word translation accuracy for similar languages.

Original: Smith et al.
Revised: Smith et al.

Original: (2017) and Artetxe et al.
Revised: (2017) and Artetxe et al.

Original: (2017) independently showed that a mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the Orthogonal Procrustes problem, Ω
Revised: (2017) independently showed that a mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the Orthogonal Procrustes problem, Ω

Original: [Equation]
Revised: [Equation]

Original: Dinu & Baroni (2015) worked on corrections to the "hubness" problem in embedding alignment, where certain word vectors may be close to many other word vectors, arising due to nonuniform density of vectors in the R d space.
Revised: Dinu & Baroni (2015) worked on corrections to the "hubness" problem in embedding alignment, where certain word vectors may be close to many other word vectors, arising due to nonuniform density of vectors in the R d space.

Original: Smith et al.
Revised: Smith et al.

Original: (2017) proposed the inverted softmax metric for inducing matchings between words in embeddings of different languages.
Revised: (2017) proposed the inverted softmax metric for inducing matchings between words in embeddings of different languages.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2016) studied the impact of normalization, centering and orthogonality constraints in linear alignment functions.
Revised: (2016) studied the impact of normalization, centering and orthogonality constraints in linear alignment functions.

Original: Jawanpuria et al.
Revised: Jawanpuria et al.

Original: (2019) presented a composition of orthogonal operators and a Mahalanobis metric of the form UBV T , U, V T ∈ O ( d ) , B (cid:31) 0 to account for observed correlations and moment differences between dimensions (Søgaard et al., 2018).
Revised: (2019) presented a composition of orthogonal operators and a Mahalanobis metric of the form UBV T , U, V T ∈ O ( d ) , B (cid:31) 0 to account for observed correlations and moment differences between dimensions (Søgaard et al., 2018).

Original: Joulin et al.
Revised: Joulin et al.

Original: (2018a) proposed an alignment based on neighborhood information to account for differences in density and shape of embeddings in their respective R d spaces.
Revised: (2018a) proposed an alignment based on neighborhood information to account for differences in density and shape of embeddings in their respective R d spaces.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2018c) outlined a framework which uniﬁes many existing alignment approaches as compositions of matrix operations such as Orthogonal mappings,
Revised: (2018c) outlined a framework which uniﬁes many existing alignment approaches as compositions of matrix operations such as Orthogonal mappings,

Original: Whitening, and Dimensionality Reduction.
Revised: Whitening, and Dimensionality Reduction.

Original: Nakashole & Flauger (2018) found that locally linear maps vary between different neighborhoods in bilingual embedding spaces which suggests that nonlinearity is beneﬁcial in global alignments.
Revised: Nakashole & Flauger (2018) found that locally linear maps vary between different neighborhoods in bilingual embedding spaces which suggests that nonlinearity is beneﬁcial in global alignments.

Original: Nakashole (2018) proposed an alignment method using neighborhood sensitive maps which shows strong performance on dissimilar language pairs.
Revised: Nakashole (2018) proposed an alignment method using neighborhood sensitive maps which shows strong performance on dissimilar language pairs.

Original: Patra et al.
Revised: Patra et al.

Original: (2019) proposed a novel hub ﬁltering method and a semi-supervised alignment approach based on distributional matching.
Revised: (2019) proposed a novel hub ﬁltering method and a semi-supervised alignment approach based on distributional matching.

Original: Mohiuddin et al.
Revised: Mohiuddin et al.

Original: (2020) learned a non-linear mapping in the latent space of two independently pre-trained autoencoders which provide strong performance on well-studied BLI tasks.
Revised: (2020) learned a non-linear mapping in the latent space of two independently pre-trained autoencoders which provide strong performance on well-studied BLI tasks.

Original: A recent method, most similar to ours, Glavaš & Vuli´c (2020) utilizes non-linear mappings to ﬁnd a translation vector for each source and target embedding using the cosine similarity and euclidean distances between nearest neighbors and corresponding translations.
Revised: A recent method, most similar to ours, Glavaš & Vuli´c (2020) utilizes non-linear mappings to ﬁnd a translation vector for each source and target embedding using the cosine similarity and euclidean distances between nearest neighbors and corresponding translations.

Original: In the unsupervised setting, where a bilingual seed dictionary is not provided, approaches using adversarial learning, distributional matching, and noisy self-supervision have been used to concurrently learn a matching and an alignment between embeddings (Cao et al., 2016;
Revised: In the unsupervised setting, where a bilingual seed dictionary is not provided, approaches using adversarial learning, distributional matching, and noisy self-supervision have been used to concurrently learn a matching and an alignment between embeddings (Cao et al., 2016;

Original: Zhang et al., 2017; Hoshen
Revised: Zhang et al., 2017; Hoshen

Original: & Wolf, 2018; Grave et al., 2019; Artetxe et al., 2017; 2018b; Alvarez-Melis & Jaakkola, 2018).
Revised: & Wolf, 2018; Grave et al., 2019; Artetxe et al., 2017; 2018b; Alvarez-Melis & Jaakkola, 2018).

Original: Discussion on unsupervised approaches is included in Appendix Section I.
Revised: Discussion on unsupervised approaches is included in Appendix Section I.

Original:  F ILTERED I NNER P RODUCT P ROJECTION (FIPP) 3. F ILTERED I NNER P RODUCT P ROJECTION O BJECTIVE
Revised:  F ILTERED I NNER P RODUCT P ROJECTION (FIPP) 3. F ILTERED I NNER P RODUCT P ROJECTION O BJECTIVE

Original: In this section, we introduce Filtered Inner Product Projection (FIPP), a method for aligning embeddings in a shared representation space.
Revised: In this section, we introduce Filtered Inner Product Projection (FIPP), a method for aligning embeddings in a shared representation space.

Original: FIPP aligns a source embedding X s ∈ R n × d 1 to a target embedding X t ∈ R m × d 2 and projects vectors in X s to ˜ X s ∈ R n × d 2 .
Revised: FIPP aligns a source embedding X s ∈ R n × d 1 to a target embedding X t ∈ R m × d 2 and projects vectors in X s to ˜ X s ∈ R n × d 2 .

Original: Let X s ∈ R c × d 1 and X t ∈ R c
Revised: Let X s ∈ R c × d 1 and X t ∈ R c

Original: ×
Revised: ×

Original: dbe the source and target embeddings for pairs in the seed dictionary D , | D | =
Revised: dbe the source and target embeddings for pairs in the seed dictionary D , | D | =

Original: c (cid:28) min ( n, m ) .
Revised: c (cid:28) min ( n, m ) .

Original: FIPP’s objective is to minimize a linear combination of a reconstruction loss, which regularizes changes in the pairwise inner products of the source embedding, and a transfer loss, which aligns the source and target embeddings on common portions of their geometries.
Revised: FIPP’s objective is to minimize a linear combination of a reconstruction loss, which regularizes changes in the pairwise inner products of the source embedding, and a transfer loss, which aligns the source and target embeddings on common portions of their geometries.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (1) where λ, (cid:15) ∈ R
Revised: (1) where λ, (cid:15) ∈ R

Original: + are tunable scalar hyperparameters whose effects are discussed in Section E, ◦ is the Hadamard product, and ∆ (cid:15) is a binary matrix discussed in 3.1.2.
Revised: + are tunable scalar hyperparameters whose effects are discussed in Section E, ◦ is the Hadamard product, and ∆ (cid:15) is a binary matrix discussed in 3.1.2.

Original: 3.1. R ECONSTRUCTION L OSS
Revised: 3.1. R ECONSTRUCTION L OSS

Original: Due to the limited, noisy supervision in our problem setting, an alignment should be regularized against overﬁtting.
Revised: Due to the limited, noisy supervision in our problem setting, an alignment should be regularized against overﬁtting.

Original: Speciﬁcally, the aligned space needs to retain a similar geometric structure to the original source embeddings; this has been enforced in previous works by ensuring that alignments are close to orthogonal mappings (Mikolov et al., 2013a; Joulin et al., 2018a; Jawanpuria et al., 2019).
Revised: Speciﬁcally, the aligned space needs to retain a similar geometric structure to the original source embeddings; this has been enforced in previous works by ensuring that alignments are close to orthogonal mappings (Mikolov et al., 2013a; Joulin et al., 2018a; Jawanpuria et al., 2019).

Original: As ˜
Revised: As ˜

Original: X s and X s can be of differing dimensionality, we check structural similarity by comparing pairwise inner products, captured by a reconstruction loss known as the PIP distance or Global Anchor Metric:
Revised: X s and X s can be of differing dimensionality, we check structural similarity by comparing pairwise inner products, captured by a reconstruction loss known as the PIP distance or Global Anchor Metric:

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose E ∈ R n × d , F ∈ R n × d are two matrices with orthonormal columns and Ω ∗ = arg min Ω ∈ O ( d ) (cid:107) E Ω − F (cid:107) F .
Revised: Suppose E ∈ R n × d , F ∈ R n × d are two matrices with orthonormal columns and Ω ∗ = arg min Ω ∈ O ( d ) (cid:107) E Ω − F (cid:107) F .

Original: It follows that (Yin et al., 2018):
Revised: It follows that (Yin et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: This metric has been used in quantifying semantic shifts and has been shown Yin et al.
Revised: This metric has been used in quantifying semantic shifts and has been shown Yin et al.

Original: (2018) to be equivalent to the residual of the Orthogonal Procrustes problem up to a small constant factor, as seen in Theorem 1.
Revised: (2018) to be equivalent to the residual of the Orthogonal Procrustes problem up to a small constant factor, as seen in Theorem 1.

Original: Note that the PIP distance is invariant to orthogonal operations such as rotations which are known to be present in unaligned embeddings.
Revised: Note that the PIP distance is invariant to orthogonal operations such as rotations which are known to be present in unaligned embeddings.

Original: 3.1. T RANSFER L OSS
Revised: 3.1. T RANSFER L OSS

Original: In aligning X s to X t , we should seek to only utilize common geometric information between the two embedding spaces.
Revised: In aligning X s to X t , we should seek to only utilize common geometric information between the two embedding spaces.

Original: We propose a simple approach, although FIPP can admit other forms of ﬁltering mechanisms, denoted as inner product ﬁltering where we only utilize pairwise distances similar in both embedding spaces as deﬁned by a threshold (cid:15) .
Revised: We propose a simple approach, although FIPP can admit other forms of ﬁltering mechanisms, denoted as inner product ﬁltering where we only utilize pairwise distances similar in both embedding spaces as deﬁned by a threshold (cid:15) .

Original: Speciﬁcally, compute a matrix ∆ (cid:15) ∈ { 0 , 1 } c × c where ∆ (cid:15)ij is an indicator on whether | X s,i X Ts,j − X t,i X Tt,j | < (cid:15) .
Revised: Speciﬁcally, compute a matrix ∆ (cid:15) ∈ { 0 , 1 } c × c where ∆ (cid:15)ij is an indicator on whether | X s,i X Ts,j − X t,i X Tt,j | < (cid:15) .

Original: In this form, (cid:15) is a hyperparameter which determines how close pairwise distances must be in the source and target embeddings to be deemed similar.
Revised: In this form, (cid:15) is a hyperparameter which determines how close pairwise distances must be in the source and target embeddings to be deemed similar.

Original: We then deﬁne a transfer loss as being the squared difference between the converted source embedding ˜ X s and target embedding X t , but only on pairs of words in
Revised: We then deﬁne a transfer loss as being the squared difference between the converted source embedding ˜ X s and target embedding X t , but only on pairs of words in

Original: K : (cid:107) ∆ (cid:15) ◦ ( ˜ X s ˜ X
Revised: K : (cid:107) ∆ (cid:15) ◦ ( ˜ X s ˜ X

Original: Ts − X t X Tt ) (cid:107) 2 F , where ◦ is the Hadamard product.
Revised: Ts − X t X Tt ) (cid:107) 2 F , where ◦ is the Hadamard product.

Original: The FIPP objective is a linear combination of the reconstruction and transfer losses.
Revised: The FIPP objective is a linear combination of the reconstruction and transfer losses.

Original: 3. A PPROXIMATE S OLUTIONS TO THE FIPP O BJECTIVE 3.2. S OLUTIONS USING L OW - RANK S EMIDEFINITE A PPROXIMATIONS
Revised: 3. A PPROXIMATE S OLUTIONS TO THE FIPP O BJECTIVE 3.2. S OLUTIONS USING L OW - RANK S EMIDEFINITE A PPROXIMATIONS

Original: Denote the Gram matrices G s (cid:44) X s X Ts , G t (cid:44)
Revised: Denote the Gram matrices G s (cid:44) X s X Ts , G t (cid:44)

Original: X t X Tt and ˜ G s (cid:44) ˜
Revised: X t X Tt and ˜ G s (cid:44) ˜

Original: X s ˜ X Ts .
Revised: X s ˜ X Ts .

Original: Lemma 2.
Revised: Lemma 2.

Original: The matrix G ∗ which minimizes the FIPP objective for a ﬁxed λ and (cid:15) has entries:
Revised: The matrix G ∗ which minimizes the FIPP objective for a ﬁxed λ and (cid:15) has entries:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For a ﬁxed λ and (cid:15) , L FIPP,λ,(cid:15) ( ˜ X s ˜ X Ts ) can be decomposed as follows:
Revised: For a ﬁxed λ and (cid:15) , L FIPP,λ,(cid:15) ( ˜ X s ˜ X Ts ) can be decomposed as follows:

Original: [Equation]
Revised: [Equation]

Original: By taking derivatives with respect to ˜ G sij , the matrix G ∗ which minimizes L FIPP,λ,(cid:15) ( · ) is:
Revised: By taking derivatives with respect to ˜ G sij , the matrix G ∗ which minimizes L FIPP,λ,(cid:15) ( · ) is:

Original: [Equation]
Revised: [Equation]

Original: We now have the matrix G ∗ ∈ R c
Revised: We now have the matrix G ∗ ∈ R c

Original: ×
Revised: ×

Original: c
Revised: c

Original: which minimizes the FIPP objective.
Revised: which minimizes the FIPP objective.

Original: However, for G ∗ to be a valid Gram matrix, it is required that G ∗ ∈
Revised: However, for G ∗ to be a valid Gram matrix, it is required that G ∗ ∈

Original: S c × c
Revised: S c × c

Original: + , the set of symmetric Positive Semideﬁnite matrices.
Revised: + , the set of symmetric Positive Semideﬁnite matrices.

Original: Additionally, to recover an ˜ X s ∈ R c × d 2 such that ˜ X s ˜ X Ts = G ∗ , we must have Rank ( G ∗ ) ≤ d 2 .
Revised: Additionally, to recover an ˜ X s ∈ R c × d 2 such that ˜ X s ˜ X Ts = G ∗ , we must have Rank ( G ∗ ) ≤ d 2 .

Original: Note that G ∗ is symmetric by construction since the set K is commutative and G s , G t are symmetric.
Revised: Note that G ∗ is symmetric by construction since the set K is commutative and G s , G t are symmetric.

Original: However, G ∗ is not necessarily positive semideﬁnite nor is it necessarily true that Rank ( G ∗ ) ≤ d 2 .
Revised: However, G ∗ is not necessarily positive semideﬁnite nor is it necessarily true that Rank ( G ∗ ) ≤ d 2 .

Original: Therefore, to recover an aligned embedding ˜ X s ∈ R c × d 2 , we perform a rank-constrained semideﬁnite approximation to ﬁnd min
Revised: Therefore, to recover an aligned embedding ˜ X s ∈ R c × d 2 , we perform a rank-constrained semideﬁnite approximation to ﬁnd min

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Let G ∗ =
Revised: Let G ∗ =

Original: Q Λ Q T be the Eigendecomposition of G ∗ .
Revised: Q Λ Q T be the Eigendecomposition of G ∗ .

Original: A matrix ˜ X s ∈ R m × d 2 which minimizes (cid:107) ˜
Revised: A matrix ˜ X s ∈ R m × d 2 which minimizes (cid:107) ˜

Original: X s ˜ X
Revised: X s ˜ X

Original: Ts − G ∗ (cid:107) F is given by (cid:80) d 2 i =1 ,λ i ≥ 

Revised: Ts − G ∗ (cid:107) F is given by (cid:80) d 2 i =1 ,λ i ≥

Original: λ 

Revised: λ

Original: i q i , where λ i and q i are the i th largest eigenvalue and corresponding eigenvector.
Revised: i q i , where λ i and q i are the i th largest eigenvalue and corresponding eigenvector.

Original: Proof.
Revised: Proof.

Original: Since G ∗ ∈ S c × c , its Eigendecomposition is G ∗ = Q Λ Q T where Q is orthonormal.
Revised: Since G ∗ ∈ S c × c , its Eigendecomposition is G ∗ = Q Λ Q T where Q is orthonormal.

Original: Let ˜ λ, ˜
Revised: Let ˜ λ, ˜

Original: q be the d 2 largest nonnegative eigenvalues in Λ and their corresponding eigenvectors; additionally, denote the complementary eigenvalues and associated eigenvectors as ˜ λ ⊥
Revised: q be the d 2 largest nonnegative eigenvalues in Λ and their corresponding eigenvectors; additionally, denote the complementary eigenvalues and associated eigenvectors as ˜ λ ⊥

Original: = Λ \ ˜ λ, ˜
Revised: = Λ \ ˜ λ, ˜

Original: q ⊥ =
Revised: q ⊥ =

Original: Q \ ˜ q .
Revised: Q \ ˜ q .

Original: Using the Eckart–Young–Mirsky Theorem for the Frobenius norm (Kishore Kumar & Schneider, 2017), note that for G ∈ S c × c + , Rank ( G ) ≤ d 2 ; (cid:107) G ∗ − G (cid:107) F ≥ (cid:107) ˜
Revised: Using the Eckart–Young–Mirsky Theorem for the Frobenius norm (Kishore Kumar & Schneider, 2017), note that for G ∈ S c × c + , Rank ( G ) ≤ d 2 ; (cid:107) G ∗ − G (cid:107) F ≥ (cid:107) ˜

Original: q ⊥ ˜ λ ⊥ ˜
Revised: q ⊥ ˜ λ ⊥ ˜

Original: q ⊥ T (cid:107) F = (cid:80) λ i ∈ ˜ λ ⊥ λ
Revised: q ⊥ T (cid:107) F = (cid:80) λ i ∈ ˜ λ ⊥ λ

Original: i and that (cid:107) G ∗ − G (cid:107) F is minimized for G =
Revised: i and that (cid:107) G ∗ − G (cid:107) F is minimized for G =

Original: ˜ q ˜ λ ˜ q T .
Revised: ˜ q ˜ λ ˜ q T .

Original: Using this result, we can recover ˜ X s : arg
Revised: Using this result, we can recover ˜ X s : arg

Original: [Equation]
Revised: [Equation]

Original: Using the above matrix approximation, we ﬁnd our aligned embedding ˜ X
Revised: Using the above matrix approximation, we ﬁnd our aligned embedding ˜ X

Original: s = (cid:80) λ i ∈ ˜ λ
Revised: s = (cid:80) λ i ∈ ˜ λ

Original: λ 12 i q i , a minimizer of (cid:107)
Revised: λ 12 i q i , a minimizer of (cid:107)

Original: ˜
Revised: ˜

Original: X s ˜ X
Revised: X s ˜ X

Original: Ts − G ∗ (cid:107) F .
Revised: Ts − G ∗ (cid:107) F .

Original: Due to the rank constraint on G , we are only interested in the d 2 largest eigenvalues and corresponding eigenvectors which incurs a complexity of O ( d 2 c 2 ) using power iteration (Panju, 2011).
Revised: Due to the rank constraint on G , we are only interested in the d 2 largest eigenvalues and corresponding eigenvectors which incurs a complexity of O ( d 2 c 2 ) using power iteration (Panju, 2011).

Original: 3.2. S OLUTIONS USING S TOCHASTIC G RADIENT D ESCENT
Revised: 3.2. S OLUTIONS USING S TOCHASTIC G RADIENT D ESCENT

Original: Alternatively, solutions to the FIPP objective can be obtained using Stochastic Gradient Descent (SGD).
Revised: Alternatively, solutions to the FIPP objective can be obtained using Stochastic Gradient Descent (SGD).

Original: This requires deﬁning a single variable ˜ X s ∈ R c × d 2 over which to optimize.
Revised: This requires deﬁning a single variable ˜ X s ∈ R c × d 2 over which to optimize.

Original: We ﬁnd that the solutions obtained after convergence of SGD are close, with respect to the Frobenius norm, to those obtained with low rank PSD approximations up to a rotation.
Revised: We ﬁnd that the solutions obtained after convergence of SGD are close, with respect to the Frobenius norm, to those obtained with low rank PSD approximations up to a rotation.

Original: However, the complexity of solving
Revised: However, the complexity of solving

Original: FIPP using SGD is O
Revised: FIPP using SGD is O

Original: ( Tc 2 ) , where T is the number of training epochs.
Revised: ( Tc 2 ) , where T is the number of training epochs.

Original: Empirically we ﬁnd T > dfor SGD convergence and, as a result, this approach incurs a complexity greater than that of low-rank semideﬁnite approximations.
Revised: Empirically we ﬁnd T > dfor SGD convergence and, as a result, this approach incurs a complexity greater than that of low-rank semideﬁnite approximations.

Original: 3. I SOTROPIC P REPROCESSING
Revised: 3. I SOTROPIC P REPROCESSING

Original: Common preprocessing steps used by previous approaches (Joulin et al., 2018a; Artetxe et al., 2018a), involve normalizing the rows of X s , X t to have an (cid:96) 2 norm of 1 and demeaning columns.
Revised: Common preprocessing steps used by previous approaches (Joulin et al., 2018a; Artetxe et al., 2018a), involve normalizing the rows of X s , X t to have an (cid:96) 2 norm of 1 and demeaning columns.

Original: The transfer loss of the FIPP objective makes direct comparisons on the Gram matrices, X s X Ts and X t X Tt , of the source and target embeddings for words in the seed dictionary.
Revised: The transfer loss of the FIPP objective makes direct comparisons on the Gram matrices, X s X Ts and X t X Tt , of the source and target embeddings for words in the seed dictionary.

Original: To reduce the inﬂuence of dimensional biases between X s and X t and ensure words are weighted equally during alignment, it is desired that X s and X t be isotropic - i.e. Z ( c ) =
Revised: To reduce the inﬂuence of dimensional biases between X s and X t and ensure words are weighted equally during alignment, it is desired that X s and X t be isotropic - i.e. Z ( c ) =

Original: (cid:80) i ∈
Revised: (cid:80) i ∈

Original: X s exp c T X
Revised: X s exp c T X

Original: s
Revised: s

Original: [ i ] is approximately constant for any unit vector c (Arora et al., 2016).
Revised: [ i ] is approximately constant for any unit vector c (Arora et al., 2016).

Original: Mu & Viswanath (2018) ﬁnd that a ﬁrst order approximation to enforce isotropic behavior is achieved by column demeaning while a second order approximation is obtained by the removal of the top PCA components.
Revised: Mu & Viswanath (2018) ﬁnd that a ﬁrst order approximation to enforce isotropic behavior is achieved by column demeaning while a second order approximation is obtained by the removal of the top PCA components.

Original: In FIPP, we apply this simple pre-processing approach by removing the top PCA component of X s and X t .
Revised: In FIPP, we apply this simple pre-processing approach by removing the top PCA component of X s and X t .

Original: Empirically, the distributions of inner products between a source and target embedding can differ substantially when not preprocessed, rendering a substandard alignment, which is discussed further in the Appendix Section G.
Revised: Empirically, the distributions of inner products between a source and target embedding can differ substantially when not preprocessed, rendering a substandard alignment, which is discussed further in the Appendix Section G.

Original:  I NFERENCE AND A LIGNMENT 4. I NFERENCE WITH L EAST S QUARES P ROJECTION
Revised:  I NFERENCE AND A LIGNMENT 4. I NFERENCE WITH L EAST S QUARES P ROJECTION

Original: To infer aligned source embeddings for words outside of the supervision dictionary, we make the assumption that source words not used for alignment should preserve their distances to those in the seed dictionary in their respective spaces, i.e., X s X Ts
Revised: To infer aligned source embeddings for words outside of the supervision dictionary, we make the assumption that source words not used for alignment should preserve their distances to those in the seed dictionary in their respective spaces, i.e., X s X Ts

Original: ≈ ˜ X s
Revised: ≈ ˜ X s

Original: ˜ X Ts .
Revised: ˜ X Ts .

Original: Using this assumption, we formulate a least squares projection (Boyd & Vandenberghe, 2004) on an overdetermined system of equations to recover ˜ X
Revised: Using this assumption, we formulate a least squares projection (Boyd & Vandenberghe, 2004) on an overdetermined system of equations to recover ˜ X

Original: Ts : ˜ X
Revised: Ts : ˜ X

Original: Ts =
Revised: Ts =

Original: ( ˜ X Ts ˜ X s ) − 1 ˜ X
Revised: ( ˜ X Ts ˜ X s ) − 1 ˜ X

Original: Ts X s X Ts .
Revised: Ts X s X Ts .

Original: 4. W EIGHTED O RTHOGONAL P ROCRUSTES
Revised: 4. W EIGHTED O RTHOGONAL P ROCRUSTES

Original: As ˜ X s ∈ R c × d 2 has been optimized only with concern for its inner products, ˜ X s must be rotated so that it’s basis matches that of X t .
Revised: As ˜ X s ∈ R c × d 2 has been optimized only with concern for its inner products, ˜ X s must be rotated so that it’s basis matches that of X t .

Original: We propose a weighted variant of the Orthogonal Procrustes solution to account for differing levels of translation uncertainty among pairs in the seed dictionary, which may arise due to polysemy, semantic shifts and translation errors.
Revised: We propose a weighted variant of the Orthogonal Procrustes solution to account for differing levels of translation uncertainty among pairs in the seed dictionary, which may arise due to polysemy, semantic shifts and translation errors.

Original: In Weighted Least Squares problems, an inverse variance-covariance weighting W is used (Strutz, 2010; Brignell et al., 2015) to account for differing levels of measurement uncertainty among samples.
Revised: In Weighted Least Squares problems, an inverse variance-covariance weighting W is used (Strutz, 2010; Brignell et al., 2015) to account for differing levels of measurement uncertainty among samples.

Original: We solve a weighted
Revised: We solve a weighted

Original: Procrustes objective, where measurement error is approximated as the transfer loss for each translation pair,
Revised: Procrustes objective, where measurement error is approximated as the transfer loss for each translation pair,

Original: [Equation]
Revised: [Equation]

Original: where O ( d 2 ) is the group of d 2 × d 2 orthogonal matrices.
Revised: where O ( d 2 ) is the group of d 2 × d 2 orthogonal matrices.

Original: The rotation Ω W is then applied to ˜ X s .
Revised: The rotation Ω W is then applied to ˜ X s .

Original:  E XPERIMENTATION
Revised:  E XPERIMENTATION

Original: In this section, we report bilingual lexicon induction results from the XLING dataset and downstream experiments performed on the MNLI Natural Language Inference and TED CLDC tasks.
Revised: In this section, we report bilingual lexicon induction results from the XLING dataset and downstream experiments performed on the MNLI Natural Language Inference and TED CLDC tasks.

Original: 5. XLING B ILINGUAL L EXICON I NDUCTION
Revised: 5. XLING B ILINGUAL L EXICON I NDUCTION

Original: The XLing BLI task dictionaries constructed by Glavaš et al.
Revised: The XLing BLI task dictionaries constructed by Glavaš et al.

Original: (2019) include all 28 pairs betweenlanguages in different language families, Croatian (HR), English (EN), Finnish (FI), French (FR),
Revised: (2019) include all 28 pairs betweenlanguages in different language families, Croatian (HR), English (EN), Finnish (FI), French (FR),

Original: German (DE), Italian (IT), Russian (RU), and Turkish (TR).
Revised: German (DE), Italian (IT), Russian (RU), and Turkish (TR).

Original: The dictionaries use the same vocabulary across languages and are constructed based on word frequency, to reduce biases known to exist in other datasets (Kementchedjhieva et al., 2019).
Revised: The dictionaries use the same vocabulary across languages and are constructed based on word frequency, to reduce biases known to exist in other datasets (Kementchedjhieva et al., 2019).

Original: We evaluate FIPP across the all language pairs using a supervision dictionary of size 5K and 1K. On 5K dictionaries, FIPP outperforms other approaches on 22 of 28 language pairs.
Revised: We evaluate FIPP across the all language pairs using a supervision dictionary of size 5K and 1K. On 5K dictionaries, FIPP outperforms other approaches on 22 of 28 language pairs.

Original: On 1K dictionaries, FIPP outperforms other approaches on 23 of language pairs when used along with a Self-Learning Framework (denoted as FIPP + SL) discussed in Appendix Section C.
Revised: On 1K dictionaries, FIPP outperforms other approaches on 23 of language pairs when used along with a Self-Learning Framework (denoted as FIPP + SL) discussed in Appendix Section C.

Original: Our code for FIPP is open-source  available on Github 1 .
Revised: Our code for FIPP is open-source and available on Github 1 .

Original: 5.1. B ILINGUAL L EXICON I NDUCTION : XLING 1K
Revised: 5.1. B ILINGUAL L EXICON I NDUCTION : XLING 1K

Original: [Table]
Revised: [Table]

Original: 5.1. B ILINGUAL L EXICON I NDUCTION : XLING 5K
Revised: 5.1. B ILINGUAL L EXICON I NDUCTION : XLING 5K

Original: [Table]
Revised: [Table]

Original: 5. D OWNSTREAM E VALUATIONS 5.2. TED CLDC - D OCUMENT C LASSIFICATION
Revised: 5. D OWNSTREAM E VALUATIONS 5.2. TED CLDC - D OCUMENT C LASSIFICATION

Original: The TED-CLDC corpus (Hermann & Blunsom, 2014) contains cross-lingual documents acrosstopics and 12 language pairs.
Revised: The TED-CLDC corpus (Hermann & Blunsom, 2014) contains cross-lingual documents acrosstopics and 12 language pairs.

Original: Following the evaluation of Glavaš et al.
Revised: Following the evaluation of Glavaš et al.

Original: (2019), a simple CNN based classiﬁer is trained and evaluated over each topic for language pairs included in our BLI evaluations (EN-DE, EN-FR, EN-IT, EN-RU
Revised: (2019), a simple CNN based classiﬁer is trained and evaluated over each topic for language pairs included in our BLI evaluations (EN-DE, EN-FR, EN-IT, EN-RU

Original: [Table]
Revised: [Table]

Original: and EN-TR).
Revised: and EN-TR).

Original: RCSLS outperforms other methods overall and on
Revised: RCSLS outperforms other methods overall and on

Original: DE, FR
Revised: DE, FR

Original: [Table]
Revised: [Table]

Original: and RU while FIPP performs best on IT and TR.
Revised: and RU while FIPP performs best on IT and TR.

Original: [Table]
Revised: [Table]

Original: 5.2. MNLI - N ATURAL L ANGUAGE I NFERENCE
Revised: 5.2. MNLI - N ATURAL L ANGUAGE I NFERENCE

Original: The multilingual XNLI corpus introduced by Conneau et al.
Revised: The multilingual XNLI corpus introduced by Conneau et al.

Original: (2018), based off the English only
Revised: (2018), based off the English only

Original: MultiNLI (Williams et al., 2018)
Revised: MultiNLI (Williams et al., 2018)

Original: [Table]
Revised: [Table]

Original: includes 5 of the 8 languages used in BLI: EN, DE, FR, RU
Revised: includes 5 of the 8 languages used in BLI: EN, DE, FR, RU

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: TR.
Revised: TR.

Original: We perform the same evaluation as Glavaš et al.
Revised: We perform the same evaluation as Glavaš et al.

Original: by training an ESIM model (Chen et al., 2017) using EN word embeddings from a shared EN-L2 embedding space for L2 ∈ {DE, FR, RU,
Revised: by training an ESIM model (Chen et al., 2017) using EN word embeddings from a shared EN-L2 embedding space for L2 ∈ {DE, FR, RU,

Original: TR}.
Revised: TR}.

Original: The trained model is then evaluated without further training on the L2 XNLI test set using
Revised: The trained model is then evaluated without further training on the L2 XNLI test set using

Original: L2 embeddings from the shared space.
Revised: L2 embeddings from the shared space.

Original: The bootstrap Procrustes approach (Glavaš et al., 2019) outperforms other methods narrowly while RCSLS performs worst despite having high BLI accuracy.
Revised: The bootstrap Procrustes approach (Glavaš et al., 2019) outperforms other methods narrowly while RCSLS performs worst despite having high BLI accuracy.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION 6. R UNTIME C OMPARISON
Revised:  D ISCUSSION 6. R UNTIME C OMPARISON

Original: [Table]
Revised: [Table]

Original: An advantage of FIPP compared to existing methods is it’s computational efﬁciency.
Revised: An advantage of FIPP compared to existing methods is it’s computational efﬁciency.

Original: We provide a runtime comparison of FIPP and the best performing unsupervised (Artetxe et al., 2018c) and supervised (Joulin et al., 2018b) alignment methods on the
Revised: We provide a runtime comparison of FIPP and the best performing unsupervised (Artetxe et al., 2018c) and supervised (Joulin et al., 2018b) alignment methods on the

Original: XLING 5K BLI tasks along with ProcB (Glavaš et al., 2019)
Revised: XLING 5K BLI tasks along with ProcB (Glavaš et al., 2019)

Original: a supervised method which performs best out of the compared approaches on 1K seed dictionaries.
Revised: a supervised method which performs best out of the compared approaches on 1K seed dictionaries.

Original: The average execution time of alignment on 3 runs of the ’EN-DE’ XLING 5K dictionary is provided in Table 4.
Revised: The average execution time of alignment on 3 runs of the ’EN-DE’ XLING 5K dictionary is provided in Table 4.

Original: The implementation used for RCSLS is from Facebook’s fastText repo 2 with default parameters and the VecMap implementation is from the author’s repo 3 with and without the ’cuda’ ﬂag.
Revised: The implementation used for RCSLS is from Facebook’s fastText repo 2 with default parameters and the VecMap implementation is from the author’s repo 3 with and without the ’cuda’ ﬂag.

Original: Proc-B is implemented from the XLING-Eval repo 4 with default parameters.
Revised: Proc-B is implemented from the XLING-Eval repo 4 with default parameters.

Original: Hardware speciﬁcations are
Revised: Hardware speciﬁcations are

Original: Nvidia GeForce GTX 1080 Ti GPUs, 12 Intel Core i7-6800K processors, and 112GB RAM.
Revised: Nvidia GeForce GTX 1080 Ti GPUs, 12 Intel Core i7-6800K processors, and 112GB RAM.

Original: 6. A LIGNMENT OF E MBEDDINGS OF D IFFERING D IMENSIONALITY 6.2. C OMPARISON OF G RAM M ATRIX S PECTRUM
Revised: 6. A LIGNMENT OF E MBEDDINGS OF D IFFERING D IMENSIONALITY 6.2. C OMPARISON OF G RAM M ATRIX S PECTRUM

Original: In this section, we compute alignments with previous methods, a modiﬁcation of Procrustes and RCSLS (Joulin et al., 2018b), and FIPP on an English Embedding ∈ R 200 to a German
Revised: In this section, we compute alignments with previous methods, a modiﬁcation of Procrustes and RCSLS (Joulin et al., 2018b), and FIPP on an English Embedding ∈ R 200 to a German

Original: Embedding ∈ R 300 .
Revised: Embedding ∈ R 300 .

Original: In Figure 2, we plot the spectrum of the Gram Matrices for the aligned embeddings from each method and the target
Revised: In Figure 2, we plot the spectrum of the Gram Matrices for the aligned embeddings from each method and the target

Original: German Embedding.
Revised: German Embedding.

Original: While the FIPP aligned embedding is isomorphic to the target German
Revised: While the FIPP aligned embedding is isomorphic to the target German

Original: Embedding ∈ R 300 , other methods produce a rank deﬁcient aligned embedding whose spectrum deviates from the target embedding.
Revised: Embedding ∈ R 300 , other methods produce a rank deﬁcient aligned embedding whose spectrum deviates from the target embedding.

Original: The null space of aligned source embeddings, for methods other than FIPP, is of dimension d 2 − d 1 .
Revised: The null space of aligned source embeddings, for methods other than FIPP, is of dimension d 2 − d 1 .

Original: We note that issues can arise when learning and transferring models on embeddings from different rank vector spaces.
Revised: We note that issues can arise when learning and transferring models on embeddings from different rank vector spaces.

Original: For regularized models transferred from the source to the target space, at least d 2 − d 1 column features of the target embedding will not be utilized.
Revised: For regularized models transferred from the source to the target space, at least d 2 − d 1 column features of the target embedding will not be utilized.

Original: Meanwhile, models transferred from the target space to the source space will exhibit bias associated with model parameters corresponding to the source embedding’s null space.
Revised: Meanwhile, models transferred from the target space to the source space will exhibit bias associated with model parameters corresponding to the source embedding’s null space.

Original: 6.2. BLI P ERFORMANCE
Revised: 6.2. BLI P ERFORMANCE

Original: [Table]
Revised: [Table]

Original: FIPP is able to align embeddings of different dimensionalities to isomorphic vector spaces unlike competing approaches (Joulin et al., 2018b; Artetxe et al., 2018c).
Revised: FIPP is able to align embeddings of different dimensionalities to isomorphic vector spaces unlike competing approaches (Joulin et al., 2018b; Artetxe et al., 2018c).

Original: We evaluate BLI performance on embeddings of different dimensionalities for the EN-DE, (English, German), language pair.
Revised: We evaluate BLI performance on embeddings of different dimensionalities for the EN-DE, (English, German), language pair.

Original: We assume that d 1 ≤ d 2 and align
Revised: We assume that d 1 ≤ d 2 and align

Original: EN embeddings with dimensions ∈ { 200 , 250 , 300 } to a DE em- bedding of dimension 300 and compare the performance of FIPP with CCA, implemented in scikit-learn 5 as an iterative estimation of partial least squares, and the best linear transform with orthonormal rows, Ω ∗ =
Revised: EN embeddings with dimensions ∈ { 200 , 250 , 300 } to a DE em- bedding of dimension 300 and compare the performance of FIPP with CCA, implemented in scikit-learn 5 as an iterative estimation of partial least squares, and the best linear transform with orthonormal rows, Ω ∗ =

Original: arg min Ω ∈ R d 1 × d 2 , ΩΩ T = I d 1 (cid:107) X
Revised: arg min Ω ∈ R d 1 × d 2 , ΩΩ T = I d 1 (cid:107) X

Original: s Ω −
Revised: s Ω −

Original: X t (cid:107) F , equivalent to the Orthogonal Procrustes solution when d 1 = d 2 .
Revised: X t (cid:107) F , equivalent to the Orthogonal Procrustes solution when d 1 = d 2 .

Original: Both the Linear and FIPP methods map X s and X t to dimension d 2 while the CCA method maps both embeddings to min ( d 1 , d 2 ) which may be undesirable.
Revised: Both the Linear and FIPP methods map X s and X t to dimension d 2 while the CCA method maps both embeddings to min ( d 1 , d 2 ) which may be undesirable.

Original: While the performance of all methods decreases as d 2 − d 1 increases, the relative performance gap between FIPP and other approaches is maintained.
Revised: While the performance of all methods decreases as d 2 − d 1 increases, the relative performance gap between FIPP and other approaches is maintained.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced Filtered Inner Product Projection (FIPP), a method for aligning multiple embeddings to a common representation space using pairwise inner product information.
Revised: In this paper, we introduced Filtered Inner Product Projection (FIPP), a method for aligning multiple embeddings to a common representation space using pairwise inner product information.

Original: FIPP accounts for semantic shifts and aligns embeddings only on common portions of their geometries.
Revised: FIPP accounts for semantic shifts and aligns embeddings only on common portions of their geometries.

Original: Unlike previous approaches, FIPP aligns embeddings to equivalent rank vector spaces regardless of their dimensions.
Revised: Unlike previous approaches, FIPP aligns embeddings to equivalent rank vector spaces regardless of their dimensions.

Original: We provide two methods for ﬁnding approximate solutions to the FIPP objective and show that it can be efﬁciently solved even in the case of large seed dictionaries.
Revised: We provide two methods for ﬁnding approximate solutions to the FIPP objective and show that it can be efﬁciently solved even in the case of large seed dictionaries.

Original: We evaluate FIPP on the task of bilingual lexicon induction using the XLING (5K) dataset and the XLING (1K) dataset, on which it achieves state-of-the-art performance on most language pairs.
Revised: We evaluate FIPP on the task of bilingual lexicon induction using the XLING (5K) dataset and the XLING (1K) dataset, on which it achieves state-of-the-art performance on most language pairs.

Original: Our method provides a novel efﬁcient approach to the problem of shared representation learning.
Revised: Our method provides a novel efﬁcient approach to the problem of shared representation learning.

Original: A F ULL BLI E XPERIMENTATION - XLING (1K) AND XLING (5K)
Revised: A F ULL BLI E XPERIMENTATION - XLING (1K) AND XLING (5K)

Original: In Table 6 below, we provide experimental results for FIPP using 1K seed dictionaries.
Revised: In Table 6 below, we provide experimental results for FIPP using 1K seed dictionaries.

Original: Unlike in case of a 5K supervision set, FIPP is outperformed by the bootstrapped Procrustes method (Glavaš et al., 2019) and the unsupervised VecMap approach (Artetxe et al., 2018c).
Revised: Unlike in case of a 5K supervision set, FIPP is outperformed by the bootstrapped Procrustes method (Glavaš et al., 2019) and the unsupervised VecMap approach (Artetxe et al., 2018c).

Original: However, the addition of a self learning framework, detailed in Section C, to FIPP (FIPP + SL) results in performance greater than compared methods albeit at the cost of close to a 8x increase in computation time, from 23s to 176s, and the requirement of a GPU.
Revised: However, the addition of a self learning framework, detailed in Section C, to FIPP (FIPP + SL) results in performance greater than compared methods albeit at the cost of close to a 8x increase in computation time, from 23s to 176s, and the requirement of a GPU.

Original: Other well performing methods for XLING 1K, Proc-B (Glavaš et al., 2019) and VecMap (Artetxe et al., 2018c), also use self-learning frameworks; further analysis is required to understand the importance of self-learning frameworks in the case of small (or no) seed dictionary.
Revised: Other well performing methods for XLING 1K, Proc-B (Glavaš et al., 2019) and VecMap (Artetxe et al., 2018c), also use self-learning frameworks; further analysis is required to understand the importance of self-learning frameworks in the case of small (or no) seed dictionary.

Original: The methods compared against were originally proposed in: VecMap (Artetxe et al., 2018c), MUSE (Lample et al., 2018), ICP (Hoshen & Wolf, 2018), CCA (Faruqui & Dyer, 2014), GWA (AlvarezMelis & Jaakkola, 2018), PROC (Mikolov et al., 2013a), PROC-B (Glavaš et al., 2019), DLV (Ruder et al., 2018), and RCSLS (Joulin et al., 2018b).
Revised: The methods compared against were originally proposed in: VecMap (Artetxe et al., 2018c), MUSE (Lample et al., 2018), ICP (Hoshen & Wolf, 2018), CCA (Faruqui & Dyer, 2014), GWA (AlvarezMelis & Jaakkola, 2018), PROC (Mikolov et al., 2013a), PROC-B (Glavaš et al., 2019), DLV (Ruder et al., 2018), and RCSLS (Joulin et al., 2018b).

Original: [Table]
Revised: [Table]

Original: 0.530 DE-HR 0.
Revised: 0.530 DE-HR 0.

Original: 0.312 DE-IT 0.
Revised: 0.312 DE-IT 0.

Original: 0.526 DE-RU 0.
Revised: 0.526 DE-RU 0.

Original: 0.368 DE-TR 0.
Revised: 0.368 DE-TR 0.

Original: EN-DE 0.
Revised: EN-DE 0.

Original: 0.568 EN-FI 0.
Revised: 0.568 EN-FI 0.

Original: 0.397 EN-FR 0.
Revised: 0.397 EN-FR 0.

Original: 0.666 EN-HR 0.
Revised: 0.666 EN-HR 0.

Original: 0.320 EN-IT 0.
Revised: 0.320 EN-IT 0.

Original: 0.638 EN-RU 0.
Revised: 0.638 EN-RU 0.

Original: 0.439 EN-TR 0.
Revised: 0.439 EN-TR 0.

Original: FI-FR 0.
Revised: FI-FR 0.

Original: 0.366 FI-HR 0.
Revised: 0.366 FI-HR 0.

Original: 0.304 FI-IT 0.
Revised: 0.304 FI-IT 0.

Original: 0.372 FI-RU 0.
Revised: 0.372 FI-RU 0.

Original: HR-FR 0.
Revised: HR-FR 0.

Original: 0.380 HR-IT 0.
Revised: 0.380 HR-IT 0.

Original: 0.389 HR-RU 0.
Revised: 0.389 HR-RU 0.

Original: IT-FR 0.
Revised: IT-FR 0.

Original: RU-FR 0.
Revised: RU-FR 0.

Original: 0.486 RU-IT 0.
Revised: 0.486 RU-IT 0.

Original: TR-FI 0.
Revised: TR-FI 0.

Original: 0.280 TR-FR 0.
Revised: 0.280 TR-FR 0.

Original: 0.342 TR-HR 0.
Revised: 0.342 TR-HR 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DLV
Revised: DLV

Original: [Table]
Revised: [Table]

Original: 0.649 EN-RU 0.
Revised: 0.649 EN-RU 0.

Original: 0.502 EN-TR 0.
Revised: 0.502 EN-TR 0.

Original: FI-FR 0.
Revised: FI-FR 0.

Original: 0.407 FI-HR 0.
Revised: 0.407 FI-HR 0.

Original: 0.335 FI-IT 0.
Revised: 0.335 FI-IT 0.

Original: 0.407 FI-RU 0.
Revised: 0.407 FI-RU 0.

Original: HR-FR 0.
Revised: HR-FR 0.

Original: 0.426 HR-IT 0.
Revised: 0.426 HR-IT 0.

Original: 0.415 HR-RU 0.
Revised: 0.415 HR-RU 0.

Original: IT-FR 0.
Revised: IT-FR 0.

Original: RU-FR 0.
Revised: RU-FR 0.

Original: 0.497 RU-IT 0.
Revised: 0.497 RU-IT 0.

Original: [Table]
Revised: [Table]

Original: B E FFECTS OF R UNNING M ULTIPLE I TERATIONS OF FIPP O PTIMIZATION
Revised: B E FFECTS OF R UNNING M ULTIPLE I TERATIONS OF FIPP O PTIMIZATION

Original: Although other alignment approaches (Joulin et al., 2018b; Artetxe et al., 2018c) run multiple iterations of their alignment objective, we ﬁnd that running multiple iterations of the FIPP Optimization does not improve performance.
Revised: Although other alignment approaches (Joulin et al., 2018b; Artetxe et al., 2018c) run multiple iterations of their alignment objective, we ﬁnd that running multiple iterations of the FIPP Optimization does not improve performance.

Original: We run between 1 and 5 iterations of the FIPP objective.
Revised: We run between 1 and 5 iterations of the FIPP objective.

Original: For 1K seed dictionaries, 26 / 28 language pairs perform best with 1 iteration while 26 / 28 language pairs perform best with 1 iteration for 5K seed dictionaries.
Revised: For 1K seed dictionaries, 26 / 28 language pairs perform best with 1 iteration while 26 / 28 language pairs perform best with 1 iteration for 5K seed dictionaries.

Original: In the case of 1K seed dictionaries, (EN, FI) with 2 iterations and (RU, FR) with 2 iterations resulted in MAP performance increases of 0.002 and 0.001.
Revised: In the case of 1K seed dictionaries, (EN, FI) with 2 iterations and (RU, FR) with 2 iterations resulted in MAP performance increases of 0.002 and 0.001.

Original: For 5K seed dictionaries, (EN, FI) with 3 iterations and (EN, FR) with 2 iterations resulted in MAP performance increases of 0.002 and 0.002.
Revised: For 5K seed dictionaries, (EN, FI) with 3 iterations and (EN, FR) with 2 iterations resulted in MAP performance increases of 0.002 and 0.002.

Original: Due to limited set of language pairs on which performance improvements were achieved, these results have not been included in Tables 1 and 6.
Revised: Due to limited set of language pairs on which performance improvements were achieved, these results have not been included in Tables 1 and 6.

Original: C S ELF -L EARNING F RAMEWORK
Revised: C S ELF -L EARNING F RAMEWORK

Original: A self-learning framework, used for augmenting the number of available training pairs without direct supervision, has been found to be effective (Glavaš et al., 2019; Artetxe et al., 2018c)
Revised: A self-learning framework, used for augmenting the number of available training pairs without direct supervision, has been found to be effective (Glavaš et al., 2019; Artetxe et al., 2018c)

Original: both in the case of small seed dictionaries or in an unsupervised setting.
Revised: both in the case of small seed dictionaries or in an unsupervised setting.

Original: We detail a self-learning framework which improves the BLI performance of FIPP in the XLING 1K setting but not in the case of 5K seed pairs.
Revised: We detail a self-learning framework which improves the BLI performance of FIPP in the XLING 1K setting but not in the case of 5K seed pairs.

Original: [Table]
Revised: [Table]

Original: Let X s ∈ R c × d 1 and X t ∈ R c × d 2 be the source and target embeddings for pairs in the seed dictionary D .
Revised: Let X s ∈ R c × d 1 and X t ∈ R c × d 2 be the source and target embeddings for pairs in the seed dictionary D .

Original: Additionally, X s ∈ R n × d 1 and X t ∈
Revised: Additionally, X s ∈ R n × d 1 and X t ∈

Original: R m × d 2 are the source and target embeddings for the entire vocabulary.
Revised: R m × d 2 are the source and target embeddings for the entire vocabulary.

Original: Assume all vectors have been normalized to have an (cid:96) 2 norm of 1 .
Revised: Assume all vectors have been normalized to have an (cid:96) 2 norm of 1 .

Original: Each source vector X s,i and target vector X t,j can be rewritten as d dimensional row vectors of inner products with their corresponding seed dictionaries:
Revised: Each source vector X s,i and target vector X t,j can be rewritten as d dimensional row vectors of inner products with their corresponding seed dictionaries:

Original: A s,i =
Revised: A s,i =

Original: X s,i X Ts ∈ R 1 × c and A s,j =
Revised: X s,i X Ts ∈ R 1 × c and A s,j =

Original: X t,j X Ts ∈ R 1 × c .
Revised: X t,j X Ts ∈ R 1 × c .

Original: For each source word i , we compute the target word j with the greatest cosine similarity, equivalent to the inner product for vectors with norm of 1 , in this d dimensional space as ( i, j ) =
Revised: For each source word i , we compute the target word j with the greatest cosine similarity, equivalent to the inner product for vectors with norm of 1 , in this d dimensional space as ( i, j ) =

Original: arg max j A s,i A
Revised: arg max j A s,i A

Original: Tt,j .
Revised: Tt,j .

Original: For
Revised: For

Original: XLING BLI 1K experiments, we ﬁnd the 14K ( i, j ) pairs with the largest cosine similarity and augment our seed dictionaries with these pairs.
Revised: XLING BLI 1K experiments, we ﬁnd the 14K ( i, j ) pairs with the largest cosine similarity and augment our seed dictionaries with these pairs.

Original: In Table 7, the top 10 translation pairings, sorted by cosine similarity, obtained using this self-learning framework are shown for the English to French (EN-FR) language pair.
Revised: In Table 7, the top 10 translation pairings, sorted by cosine similarity, obtained using this self-learning framework are shown for the English to French (EN-FR) language pair.

Original: D C OMPARISON OF FIPP SOLUTION TO O RTHOGONAL A LIGNMENTS
Revised: D C OMPARISON OF FIPP SOLUTION TO O RTHOGONAL A LIGNMENTS

Original: In this section, we conduct experimentation to quantify the degree to which the FIPP alignment differs from an orthogonal solution and compare performance on monolingual tasks before and after
Revised: In this section, we conduct experimentation to quantify the degree to which the FIPP alignment differs from an orthogonal solution and compare performance on monolingual tasks before and after

Original: FIPP alignment.
Revised: FIPP alignment.

Original: D. D EVIATION OF FIPP FROM THE CLOSEST O RTHOGONAL SOLUTION
Revised: D. D EVIATION OF FIPP FROM THE CLOSEST O RTHOGONAL SOLUTION

Original: For each language pair, we quantify the deviation of FIPP from an Orthogonal solution by ﬁrst calculating the FIPP alignment before rotation, ˜ X s , on the seed dictionary.
Revised: For each language pair, we quantify the deviation of FIPP from an Orthogonal solution by ﬁrst calculating the FIPP alignment before rotation, ˜ X s , on the seed dictionary.

Original: We then compute the relative deviation of the FIPP alignment with the closest orthogonal alignment on the original embedding, X s .
Revised: We then compute the relative deviation of the FIPP alignment with the closest orthogonal alignment on the original embedding, X s .

Original: This is equal to D = (cid:107) ˜
Revised: This is equal to D = (cid:107) ˜

Original: X s − Ω ∗ X s (cid:107) F (cid:107) X s (cid:107) F where Ω ∗ =
Revised: X s − Ω ∗ X s (cid:107) F (cid:107) X s (cid:107) F where Ω ∗ =

Original: arg min Ω ∈ O ( d 2 ) (cid:107)
Revised: arg min Ω ∈ O ( d 2 ) (cid:107)

Original: X s Ω − ˜ X s (cid:107) F .
Revised: X s Ω − ˜ X s (cid:107) F .

Original: The average of these deviations for 1K seed dictionaries is 0 .
Revised: The average of these deviations for 1K seed dictionaries is 0 .

Original: 292 and for 5K seed dictionaries is 0 . 115 .
Revised: 292 and for 5K seed dictionaries is 0 . 115 .

Original: Additionally, the 3 language pairs with the largest and smallest deviations from orthogonal solutions are presented in the Table below.
Revised: Additionally, the 3 language pairs with the largest and smallest deviations from orthogonal solutions are presented in the Table below.

Original: We ﬁnd that in most cases, small deviations from orthogonal solutions are observed between languages in the same language family (i.e. Indo-European -> IndoEuropean) while those in different Language families tend to have larger deviations (i.e. Turkic ->
Revised: We ﬁnd that in most cases, small deviations from orthogonal solutions are observed between languages in the same language family (i.e. Indo-European -> IndoEuropean) while those in different Language families tend to have larger deviations (i.e. Turkic ->

Original: Indo-European).
Revised: Indo-European).

Original: A notable exception to this observation is English and Finnish which belong to different language families, Indo-European and Uralic respectively, yet have small deviations in their
Revised: A notable exception to this observation is English and Finnish which belong to different language families, Indo-European and Uralic respectively, yet have small deviations in their

Original: FIPP solution compared to an orthogonal alignment.
Revised: FIPP solution compared to an orthogonal alignment.

Original: [Table]
Revised: [Table]

Original: D. E FFECT OF I NNER P RODUCT F ILTERING ON W ORD - LEVEL A LIGNMENT
Revised: D. E FFECT OF I NNER P RODUCT F ILTERING ON W ORD - LEVEL A LIGNMENT

Original: In FIPP, Inner Product Filtering is used to ﬁnd common geometric information by comparing pairwise distances between a source and target language.
Revised: In FIPP, Inner Product Filtering is used to ﬁnd common geometric information by comparing pairwise distances between a source and target language.

Original: To illustrate this step with translation word pairs, in the Table below we show the 5 words with the largest and smallest fraction of zeros,
Revised: To illustrate this step with translation word pairs, in the Table below we show the 5 words with the largest and smallest fraction of zeros,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. the "least and most ﬁltered", in the binary ﬁltering matrix ∆ during alignment between English (En) and Italian (It).
Revised: e. the "least and most ﬁltered", in the binary ﬁltering matrix ∆ during alignment between English (En) and Italian (It).

Original: The words which are least ﬁltered tend to have an individual word sense, i.e. proper nouns, while those which are most ﬁltered are somewhat ambiguous translations.
Revised: The words which are least ﬁltered tend to have an individual word sense, i.e. proper nouns, while those which are most ﬁltered are somewhat ambiguous translations.

Original: For instance, while the English word "securing" can be translated to the Italian word "ﬁssagio", depending on the context the Italian words "garantire", "assicurare" or "ﬁssare" may be more appropriate.
Revised: For instance, while the English word "securing" can be translated to the Italian word "ﬁssagio", depending on the context the Italian words "garantire", "assicurare" or "ﬁssare" may be more appropriate.

Original: [Table]
Revised: [Table]

Original: D. M ONOLINGUAL T ASK P ERFORMANCE OF A LIGNED E MBEDDINGS
Revised: D. M ONOLINGUAL T ASK P ERFORMANCE OF A LIGNED E MBEDDINGS

Original: [Table]
Revised: [Table]

Original: As FIPP does not perform an orthogonal transform, it modiﬁes the inner products of word vectors in the source embedding which can impact performance on monolingual task accuracy.
Revised: As FIPP does not perform an orthogonal transform, it modiﬁes the inner products of word vectors in the source embedding which can impact performance on monolingual task accuracy.

Original: We evaluate the aligned embedding learned using
Revised: We evaluate the aligned embedding learned using

Original: FIPP, ˜ X s , on monolingual word analogy tasks and compare these results to the original fastText embeddings X s .
Revised: FIPP, ˜ X s , on monolingual word analogy tasks and compare these results to the original fastText embeddings X s .

Original: In Table 3, we compare monolingual English word analogy results for English embeddings ˜ X s which have been aligned to a
Revised: In Table 3, we compare monolingual English word analogy results for English embeddings ˜ X s which have been aligned to a

Original: Turkish target embedding using FIPP.
Revised: Turkish target embedding using FIPP.

Original: Evaluation of the aligned and original source embeddings on multiple English word analogy experiments show that aligned FIPP embeddings retain performance on monolingual analogy tasks.
Revised: Evaluation of the aligned and original source embeddings on multiple English word analogy experiments show that aligned FIPP embeddings retain performance on monolingual analogy tasks.

Original: E E FFECT OF HYPERPARAMETERS ON BLI PERFORMANCE
Revised: E E FFECT OF HYPERPARAMETERS ON BLI PERFORMANCE

Original: In our experiments, we tune the hyperparameters (cid:15) and λ which signify the level of discrimination in the inner product ﬁltering step and the weight of the transfer loss respectively.
Revised: In our experiments, we tune the hyperparameters (cid:15) and λ which signify the level of discrimination in the inner product ﬁltering step and the weight of the transfer loss respectively.

Original: When tuning, we account for the sparsity of ∆ (cid:15) by scaling λ in the transfer loss by γ = c 2 NNZ (∆ (cid:15) ) where NNZ (∆ (cid:15) ) is the number of nonzeroes in ∆ (cid:15) .
Revised: When tuning, we account for the sparsity of ∆ (cid:15) by scaling λ in the transfer loss by γ = c 2 NNZ (∆ (cid:15) ) where NNZ (∆ (cid:15) ) is the number of nonzeroes in ∆ (cid:15) .

Original: Values of (cid:15) used in tuning were [0 . 01 , 0 .
Revised: Values of (cid:15) used in tuning were [0 . 01 , 0 .

Original: 10 , 0 . 15] and scaled values of λ used in tuning were [0 . 25 , 0 . 5 , 0 .
Revised: 10 , 0 . 15] and scaled values of λ used in tuning were [0 . 25 , 0 . 5 , 0 .

Original: Values of ( (cid:15), λ ) which are close to one another result in similar performance and results are deterministic across reruns of the same experiment.
Revised: Values of ( (cid:15), λ ) which are close to one another result in similar performance and results are deterministic across reruns of the same experiment.

Original: As no validation set is provided, hyperparameters are tuned by holding out 20 % of the training set.
Revised: As no validation set is provided, hyperparameters are tuned by holding out 20 % of the training set.

Original: F D IFFERENCE IN S OLUTIONS OBTAINED USING L OW R ANK A PPROXIMATIONS AND SGD
Revised: F D IFFERENCE IN S OLUTIONS OBTAINED USING L OW R ANK A PPROXIMATIONS AND SGD

Original: As detailed in Section 3, solutions to FIPP can be calculated either using Low Rank Approximations or Stochastic Gradient Descent (SGD).
Revised: As detailed in Section 3, solutions to FIPP can be calculated either using Low Rank Approximations or Stochastic Gradient Descent (SGD).

Original: In this section, we show the error on the FIPP objective for SGD trained over 10,000 epochs on alignment of a Finnish (FI) embedding to a
Revised: In this section, we show the error on the FIPP objective for SGD trained over 10,000 epochs on alignment of a Finnish (FI) embedding to a

Original: French (FR) embedding.
Revised: French (FR) embedding.

Original: The Adam (Kingma & Ba, 2015) optimizer is used with a learning rate of 1 e − 3 and the variable being optimized ˜
Revised: The Adam (Kingma & Ba, 2015) optimizer is used with a learning rate of 1 e − 3 and the variable being optimized ˜

Original: X SGDs is initialized to the original Finnish embedding X s .
Revised: X SGDs is initialized to the original Finnish embedding X s .

Original: We ﬁnd that the SGD solution ˜
Revised: We ﬁnd that the SGD solution ˜

Original: X SGDs approaches the error of the Low Rank
Revised: X SGDs approaches the error of the Low Rank

Original: Approximation ˜ X LRAs , which is the global minima of the FIPP objective, as shown in the Figure below but is not equivalent.
Revised: Approximation ˜ X LRAs , which is the global minima of the FIPP objective, as shown in the Figure below but is not equivalent.

Original: While the deviation between SGD and the Low Rank Approximation,
Revised: While the deviation between SGD and the Low Rank Approximation,

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: 041 , is smaller than the deviation between the original embedding and the Low Rank Approximation,
Revised: 041 , is smaller than the deviation between the original embedding and the Low Rank Approximation,

Original: D Orig,LRA =
Revised: D Orig,LRA =

Original: [Equation]
Revised: [Equation]

Original: LRAs
Revised: LRAs

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: 343 we note that the solutions are close but not equivalent.
Revised: 343 we note that the solutions are close but not equivalent.

Original: G E FFECTS OF P REPROCESSING ON I NNER P RODUCT D ISTRIBUTIONS
Revised: G E FFECTS OF P REPROCESSING ON I NNER P RODUCT D ISTRIBUTIONS

Original: We plot the distributions of inner products, entries of X s X Ts and X t X Tt , for English (En) and
Revised: We plot the distributions of inner products, entries of X s X Ts and X t X Tt , for English (En) and

Original: Turkish (Tr) words in the XLING 5K training dictionary before and after preprocessing in Figure below.
Revised: Turkish (Tr) words in the XLING 5K training dictionary before and after preprocessing in Figure below.

Original: All embeddings used in experimentation are fastText word vectors trained on Wikipedia (Bojanowski et al., 2017).
Revised: All embeddings used in experimentation are fastText word vectors trained on Wikipedia (Bojanowski et al., 2017).

Original: Since inner products between X s and X t are compared directly, the isotropic preprocessing utilized in FIPP is necessary for removing biases caused by variations in scaling, shifts, and point density across embedding spaces.
Revised: Since inner products between X s and X t are compared directly, the isotropic preprocessing utilized in FIPP is necessary for removing biases caused by variations in scaling, shifts, and point density across embedding spaces.

Original: H C OMPLEXITY A NALYSIS
Revised: H C OMPLEXITY A NALYSIS

Original: In this section, we provide the computational and space complexity of the FIPP method as described in the paper for computing ˜ X s .
Revised: In this section, we provide the computational and space complexity of the FIPP method as described in the paper for computing ˜ X s .

Original: We split the complexity for each step in the method.
Revised: We split the complexity for each step in the method.

Original: We leave out steps (i.e. preprocessing) which do not contribute signiﬁcantly to the runtime or memory footprint.
Revised: We leave out steps (i.e. preprocessing) which do not contribute signiﬁcantly to the runtime or memory footprint.

Original: The computational complexity of matrix multiplication between two matrices A 1 A 2 , where
Revised: The computational complexity of matrix multiplication between two matrices A 1 A 2 , where

Original: A 1 ∈ R m × n A 2 ∈ R n × p , is denoted as MM ( m, n, p ) which is upper bounded by 2 mnp operations.
Revised: A 1 ∈ R m × n A 2 ∈ R n × p , is denoted as MM ( m, n, p ) which is upper bounded by 2 mnp operations.

Original: H. FIPP O PTIMIZATION
Revised: H. FIPP O PTIMIZATION

Original: The complexity for solving the FIPP objective is detailed below:
Revised: The complexity for solving the FIPP objective is detailed below:

Original: [Equation]
Revised: [Equation]

Original: H. SVD A LIGNMENT AND L EAST S QUARE P ROJECTION
Revised: H. SVD A LIGNMENT AND L EAST S QUARE P ROJECTION

Original: The complexity of the alignment using the Orthogonal Procrustes solution and the Least Squares
Revised: The complexity of the alignment using the Orthogonal Procrustes solution and the Least Squares

Original: Projection is as follows:
Revised: Projection is as follows:

Original: [Equation]
Revised: [Equation]

Original: H. D ISCUSSION
Revised: H. D ISCUSSION

Original: In performing our analysis, we note that the majority of operations performed are quadratic in the training set size c .
Revised: In performing our analysis, we note that the majority of operations performed are quadratic in the training set size c .

Original: While we incur a time complexity of O ( c 3 ) during our Least Squares Projection due to the matrix inversion in the normal equation, this inversion is a one time cost.
Revised: While we incur a time complexity of O ( c 3 ) during our Least Squares Projection due to the matrix inversion in the normal equation, this inversion is a one time cost.

Original: The space complexity of FIPP is O ( c 2 ) which is tractable as c is at most 5 K .
Revised: The space complexity of FIPP is O ( c 2 ) which is tractable as c is at most 5 K .

Original: Empirically, FIPP is fast to compute taking less than 30 seconds for a seed dictionary of size 5 K which is more efﬁcient than competing methods.
Revised: Empirically, FIPP is fast to compute taking less than 30 seconds for a seed dictionary of size 5 K which is more efﬁcient than competing methods.

Original: I R ELATED W ORKS : U NSUPERVISED A LIGNMENT M ETHODS
Revised: I R ELATED W ORKS : U NSUPERVISED A LIGNMENT M ETHODS

Original: Cao et al.
Revised: Cao et al.

Original: (2016) studied aligning the ﬁrst two moments of sets of embeddings under Gaussian distribution assumptions.
Revised: (2016) studied aligning the ﬁrst two moments of sets of embeddings under Gaussian distribution assumptions.

Original: In Zhang et al.
Revised: In Zhang et al.

Original: (2017), an alignment between embeddings of different languages is found by matching distributions using an adversarial autoencoder with an orthogonal regularizer.
Revised: (2017), an alignment between embeddings of different languages is found by matching distributions using an adversarial autoencoder with an orthogonal regularizer.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2017) proposes an alignment approach which jointly bootstraps a small seed dictionary and learns an alignment in a self-learning framework.
Revised: (2017) proposes an alignment approach which jointly bootstraps a small seed dictionary and learns an alignment in a self-learning framework.

Original: Hoshen & Wolf (2018) ﬁrst projects embeddings to a subspace spanned by the top p principle components and then learns an alignment by matching the distributions of the projected embeddings.
Revised: Hoshen & Wolf (2018) ﬁrst projects embeddings to a subspace spanned by the top p principle components and then learns an alignment by matching the distributions of the projected embeddings.

Original: Grave et al.
Revised: Grave et al.

Original: (2019) proposes an unsupervised variant to the Orthogonal Procrustes alignment which jointly learns an orthogonal transform Ω ∈ R d × d and an assignment matrix P ∈ { 0 , 1 } n × n to minimize the Wasserstein distance between the embeddings subject to an unknown rotation.
Revised: (2019) proposes an unsupervised variant to the Orthogonal Procrustes alignment which jointly learns an orthogonal transform Ω ∈ R d × d and an assignment matrix P ∈ { 0 , 1 } n × n to minimize the Wasserstein distance between the embeddings subject to an unknown rotation.

Original: Three approaches utilize the Gram matrices of embeddings in computing alignment initializations and matchings.
Revised: Three approaches utilize the Gram matrices of embeddings in computing alignment initializations and matchings.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2018b) studied the alignment of dissimilar language pairs using a Gram matrix based initialization and robust selflearning.
Revised: (2018b) studied the alignment of dissimilar language pairs using a Gram matrix based initialization and robust selflearning.

Original: Alvarez-Melis & Jaakkola (2018) proposed an Optimal Transport based approach using the
Revised: Alvarez-Melis & Jaakkola (2018) proposed an Optimal Transport based approach using the

Original: Gromov-Wasserstein distance, GW ( C, C (cid:48) , p, q ) =
Revised: Gromov-Wasserstein distance, GW ( C, C (cid:48) , p, q ) =

Original: min Γ ∈ Π( p,q ) (cid:80) i,j,k,l L ( C ik , C (cid:48) jl )Γ ij Γ kl where
Revised: min Γ ∈ Π( p,q ) (cid:80) i,j,k,l L ( C ik , C (cid:48) jl )Γ ij Γ kl where

Original: C, C (cid:48) are Gram matrices for normalized embeddings and Γ is an assignment.
Revised: C, C (cid:48) are Gram matrices for normalized embeddings and Γ is an assignment.

Original: Aldarmaki et al.
Revised: Aldarmaki et al.

Original: (2018) learns a unsupervised linear mapping between a source and target language with a loss at each iteration equal to the sum of squares of proposed source and target Gram matrices.
Revised: (2018) learns a unsupervised linear mapping between a source and target language with a loss at each iteration equal to the sum of squares of proposed source and target Gram matrices.

Original: J A BLATION S TUDIES J. A GGREGATE MAP PERFORMANCE FOR BLI TASKS
Revised: J A BLATION S TUDIES J. A GGREGATE MAP PERFORMANCE FOR BLI TASKS

Original: [Table]
Revised: [Table]

Original: We provide an ablation study to quantify improvements associated with the three modiﬁcations of our alignment approach compared to a standard
Revised: We provide an ablation study to quantify improvements associated with the three modiﬁcations of our alignment approach compared to a standard

Original: Procrustes alignment: (i) isotropic pre-processing (IP), (ii) inner product ﬁltering (IPF) and (iii) weighted procrustes objectives (WP), on the
Revised: Procrustes alignment: (i) isotropic pre-processing (IP), (ii) inner product ﬁltering (IPF) and (iii) weighted procrustes objectives (WP), on the

Original: XLING 1K and 5K BLI tasks.
Revised: XLING 1K and 5K BLI tasks.

Original: For seed dictionaries of size 1K, improvements associated with each portion of FIPP are approximately the same while for seed dictionaries of size 5K, a larger improvement is obtained due to isotropic pre-processing than inner product ﬁltering or weighted procrustes alignment.
Revised: For seed dictionaries of size 1K, improvements associated with each portion of FIPP are approximately the same while for seed dictionaries of size 5K, a larger improvement is obtained due to isotropic pre-processing than inner product ﬁltering or weighted procrustes alignment.

Original: J. P REPROCESSING A BLATION
Revised: J. P REPROCESSING A BLATION

Original: We perform an ablation study to understand the impact of preprocessing on XLING 1K and 5K BLI performance both on Procrustes and FIPP.
Revised: We perform an ablation study to understand the impact of preprocessing on XLING 1K and 5K BLI performance both on Procrustes and FIPP.

Original: Additionally, we compare the isotropic preprocessing used in our previous experimentation with iterative normalization, a well performing preprocessing method proposed by Zhang et al.
Revised: Additionally, we compare the isotropic preprocessing used in our previous experimentation with iterative normalization, a well performing preprocessing method proposed by Zhang et al.

Original: For training dictionaries of size 1K, iterative normalization and isotropic preprocessing before running procrustes result in equivalent aggregate MAP performance of 0.316.
Revised: For training dictionaries of size 1K, iterative normalization and isotropic preprocessing before running procrustes result in equivalent aggregate MAP performance of 0.316.

Original: We ﬁnd that iterative normalization and isotropic preprocessing each achieve better performance on 11 of 28 and 12 oflanguage pairs respectively.
Revised: We ﬁnd that iterative normalization and isotropic preprocessing each achieve better performance on 11 of 28 and 12 oflanguage pairs respectively.

Original: Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for 1K training dictionaries (0.344) when compared to iterative normalization (0.331).
Revised: Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for 1K training dictionaries (0.344) when compared to iterative normalization (0.331).

Original: In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.
Revised: In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.

Original: Method
Revised: Method

Original: [Table]
Revised: [Table]

Original: DE-FI 0.
Revised: DE-FI 0.

Original: DE
Revised: DE

Original: [Table]
Revised: [Table]

Original: FR 0.
Revised: FR 0.

Original: DE
Revised: DE

Original: [Table]
Revised: [Table]

Original: HR 0.
Revised: HR 0.

Original: DE-IT 0.
Revised: DE-IT 0.

Original: DE
Revised: DE

Original: [Table]
Revised: [Table]

Original: RU 0.
Revised: RU 0.

Original: DE-TR 0.
Revised: DE-TR 0.

Original: EN
Revised: EN

Original: [Table]
Revised: [Table]

Original: DE 0.
Revised: DE 0.

Original: EN-FI 0.
Revised: EN-FI 0.

Original: EN
Revised: EN

Original: [Table]
Revised: [Table]

Original: FR 0.
Revised: FR 0.

Original: EN-HR 0.
Revised: EN-HR 0.

Original: EN-IT 0.
Revised: EN-IT 0.

Original: EN-RU 0.
Revised: EN-RU 0.

Original: EN-TR 0.
Revised: EN-TR 0.

Original: FI-FR 0.
Revised: FI-FR 0.

Original: FI
Revised: FI

Original: [Table]
Revised: [Table]

Original: HR 0.
Revised: HR 0.

Original: FI-IT 0.
Revised: FI-IT 0.

Original: FI-RU 0.
Revised: FI-RU 0.

Original: HR-FR 0.
Revised: HR-FR 0.

Original: HR-IT 0.
Revised: HR-IT 0.

Original: HR-RU 0.
Revised: HR-RU 0.

Original: [Table]
Revised: [Table]

Original: For training dictionaries of size 5K, iterative normalization and isotropic preprocessing before running procrustes result in comparable aggregate MAP performances of 0.422 and 0.424 respectively.
Revised: For training dictionaries of size 5K, iterative normalization and isotropic preprocessing before running procrustes result in comparable aggregate MAP performances of 0.422 and 0.424 respectively.

Original: We ﬁnd that iterative normalization and isotropic preprocessing each achieve better performance on 8 of 28 and 16 of 28 language pairs respectively.
Revised: We ﬁnd that iterative normalization and isotropic preprocessing each achieve better performance on 8 of 28 and 16 of 28 language pairs respectively.

Original: Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for 5K training dictionaries (0.442) when compared to iterative normalization (0.425).
Revised: Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for 5K training dictionaries (0.442) when compared to iterative normalization (0.425).

Original: In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.
Revised: In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.

Original: [Table]
Revised: [Table]

Original: DE-RU 0.
Revised: DE-RU 0.

Original: DE-TR 0.
Revised: DE-TR 0.

Original: EN-DE 0.
Revised: EN-DE 0.

Original: EN-FI 0. EN-FR 0.
Revised: EN-FI 0. EN-FR 0.

Original: EN-HR 0.
Revised: EN-HR 0.

Original: EN-IT 0.
Revised: EN-IT 0.

Original: EN-RU 0.
Revised: EN-RU 0.

Original: EN-TR 0.
Revised: EN-TR 0.

Original: FI-FR 0.
Revised: FI-FR 0.

Original: FI-HR 0.
Revised: FI-HR 0.

Original: FI-IT 0.
Revised: FI-IT 0.

Original: FI-RU 0.
Revised: FI-RU 0.

Original: HR-FR 0.
Revised: HR-FR 0.

Original: HR-IT 0.
Revised: HR-IT 0.

Original: HR-RU 0.
Revised: HR-RU 0.

Original: IT-FR 0.
Revised: IT-FR 0.

Original: RU-FR 0.
Revised: RU-FR 0.

Original: RU-IT 0.
Revised: RU-IT 0.

Original: TR-FI 0. TR-FR 0.
Revised: TR-FI 0. TR-FR 0.

Original: TR-HR 0.
Revised: TR-HR 0.

Original: TR-IT 0.
Revised: TR-IT 0.

Original: [Table]
Revised: [Table]

Original: J. W EIGHTED P ROCRUSTES A BLATION
Revised: J. W EIGHTED P ROCRUSTES A BLATION

Original: In order to measure the effect of a Weighted Procrustes rotation on BLI performance, we perform an ablation on both the 1K and 5K XLING BLI datasets against the standard Procrustes formulation.
Revised: In order to measure the effect of a Weighted Procrustes rotation on BLI performance, we perform an ablation on both the 1K and 5K XLING BLI datasets against the standard Procrustes formulation.

Original: For training dictionaries of size 1K, weighted procrustes achieves a marginally better aggregate
Revised: For training dictionaries of size 1K, weighted procrustes achieves a marginally better aggregate

Original: MAP performance when compared to standard procrustes - 0.344 vs 0.336 respectively.
Revised: MAP performance when compared to standard procrustes - 0.344 vs 0.336 respectively.

Original: In 27 oflanguage pairs, weighted procrustes provides improved MAP performance over standard procrustes.
Revised: In 27 oflanguage pairs, weighted procrustes provides improved MAP performance over standard procrustes.

Original: With training dictionaries of size 5K, weighted procrustes achieves a marginally better aggregate
Revised: With training dictionaries of size 5K, weighted procrustes achieves a marginally better aggregate

Original: MAP performance when compared to standard procrustes - 0.442 vs 0.440 respectively.
Revised: MAP performance when compared to standard procrustes - 0.442 vs 0.440 respectively.

Original: In 21 oflanguage pairs, weighted procrustes provides improved MAP performance over standard procrustes.
Revised: In 21 oflanguage pairs, weighted procrustes provides improved MAP performance over standard procrustes.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: K BLI P ERFORMANCE WITH A LTERNATIVE R ETRIEVAL C RITERIA
Revised: K BLI P ERFORMANCE WITH A LTERNATIVE R ETRIEVAL C RITERIA

Original: In order to measure the effect on BLI performance of different retrieval criteria, we perform experimentation using CSLS and Nearest Neighbors on both the 1K and 5K XLING BLI datasets.
Revised: In order to measure the effect on BLI performance of different retrieval criteria, we perform experimentation using CSLS and Nearest Neighbors on both the 1K and 5K XLING BLI datasets.

Original: We ﬁnd that the CSLS retrieval criterion provides signiﬁcant performance improvements, compared to Nearest Neighbors search, both for Procrustes and FIPP on both 1K and 5K training dictionaries.
Revised: We ﬁnd that the CSLS retrieval criterion provides signiﬁcant performance improvements, compared to Nearest Neighbors search, both for Procrustes and FIPP on both 1K and 5K training dictionaries.

Original: For 1K training dictionaries, CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.051 and 0.043 for Procrustes and FIPP respectively.
Revised: For 1K training dictionaries, CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.051 and 0.043 for Procrustes and FIPP respectively.

Original: In the case of 5K training dictionaries,
Revised: In the case of 5K training dictionaries,

Original: CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.049 and 0.031 for
Revised: CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.049 and 0.031 for

Original: Procrustes and FIPP respectively.
Revised: Procrustes and FIPP respectively.

Original: Method
Revised: Method

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DE-FI 0.
Revised: DE-FI 0.

Original: DE-FR 0.
Revised: DE-FR 0.

Original: DE-HR 0.
Revised: DE-HR 0.

Original: DE-IT 0.
Revised: DE-IT 0.

Original: DE-RU 0.
Revised: DE-RU 0.

Original: DE-TR 0.
Revised: DE-TR 0.

Original: EN-DE 0.
Revised: EN-DE 0.

Original: EN-FI 0. EN-FR 0.
Revised: EN-FI 0. EN-FR 0.

Original: EN-HR 0.
Revised: EN-HR 0.

Original: EN-IT 0.
Revised: EN-IT 0.

Original: EN-RU 0.
Revised: EN-RU 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DE-IT 0.
Revised: DE-IT 0.

Original: DE-RU 0.
Revised: DE-RU 0.

Original: DE-TR 0.
Revised: DE-TR 0.

Original: EN-DE 0.
Revised: EN-DE 0.

Original: [Table]
Revised: [Table]