Original: N ORMALIZATION OF L ANGUAGE E MBEDDINGS FOR C ROSS -L INGUAL A LIGNMENT
Revised: N ORMALIZATION OF L ANGUAGE E MBEDDINGS FOR C ROSS -L INGUAL A LIGNMENT

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning a good transfer function to map the word vectors from two languages into a shared cross-lingual word vector space plays a crucial role in cross-lingual NLP.
Revised: Learning a good transfer function to map the word vectors from two languages into a shared cross-lingual word vector space plays a crucial role in cross-lingual NLP.

Original: It is useful in translation tasks and important in allowing complex models built on a high-resource language like English to be directly applied on an aligned low resource language.
Revised: It is useful in translation tasks and important in allowing complex models built on a high-resource language like English to be directly applied on an aligned low resource language.

Original: While Procrustes and other techniques can align language models with some success, it has recently been identified that structural differences (for instance, due to differing word frequency) create different profiles for various monolingual embedding.
Revised: While Procrustes and other techniques can align language models with some success, it has recently been identified that structural differences (for instance, due to differing word frequency) create different profiles for various monolingual embedding.

Original: When these profiles differ across languages, it correlates with how well languages can align and their performance on cross-lingual downstream tasks.
Revised: When these profiles differ across languages, it correlates with how well languages can align and their performance on cross-lingual downstream tasks.

Original: In this work, we develop a very general language embedding normalization procedure, building and subsuming various previous approaches, which removes these structural profiles across languages without destroying their intrinsic meaning.
Revised: In this work, we develop a very general language embedding normalization procedure, building and subsuming various previous approaches, which removes these structural profiles across languages without destroying their intrinsic meaning.

Original: We demonstrate that meaning is retained and alignment is improved on similarity, translation, and cross-language classification tasks.
Revised: We demonstrate that meaning is retained and alignment is improved on similarity, translation, and cross-language classification tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The standard multilingual NLP approaches typically do not jointly learn a single embedding, since words tend to cluster by language, and thus are not useful for translation and cross-lingual learning tasks.
Revised: The standard multilingual NLP approaches typically do not jointly learn a single embedding, since words tend to cluster by language, and thus are not useful for translation and cross-lingual learning tasks.

Original: Rather, after learning individual embeddings, the usual approach is to map word vectors from multiple languages into a shared cross-lingual word vector space (Glava ˇ s et al., 2019).
Revised: Rather, after learning individual embeddings, the usual approach is to map word vectors from multiple languages into a shared cross-lingual word vector space (Glava ˇ s et al., 2019).

Original: This shared space creates a cross-lingual word embedding (CLWE) (Karan et al., 2020; Wang et al., 2019).
Revised: This shared space creates a cross-lingual word embedding (CLWE) (Karan et al., 2020; Wang et al., 2019).

Original: These serve as a valuable tool for transferring data across different languages, understanding crosslinguistic differences, and cross-lingual transfer for downstream tasks, such as direct translation (Gouws et al., 2015; Heyman et al., 2017; Lample et al., 2018), cross-lingual information retrieval (Vulic & Moens, 2015), cross-lingual document classification (Klementiev et al., 2012), and crosslingual dependency parsing (Guo et al., 2015; Ahmad et al., 2019).
Revised: These serve as a valuable tool for transferring data across different languages, understanding crosslinguistic differences, and cross-lingual transfer for downstream tasks, such as direct translation (Gouws et al., 2015; Heyman et al., 2017; Lample et al., 2018), cross-lingual information retrieval (Vulic & Moens, 2015), cross-lingual document classification (Klementiev et al., 2012), and crosslingual dependency parsing (Guo et al., 2015; Ahmad et al., 2019).

Original: A common element of almost all CLWE methods is the use of a rigid, orthogonal transformation mapping one embedding onto another so they inhabit a shared linguistic space.
Revised: A common element of almost all CLWE methods is the use of a rigid, orthogonal transformation mapping one embedding onto another so they inhabit a shared linguistic space.

Original: An orthogonal transformation is a special class of transformations that can be interpreted as the space of (in our case, high-dimensional) rotations around the origin, and allowing a mirror flip.
Revised: An orthogonal transformation is a special class of transformations that can be interpreted as the space of (in our case, high-dimensional) rotations around the origin, and allowing a mirror flip.

Original: This family of transformations preserves (a) linear and (b) angular properties.
Revised: This family of transformations preserves (a) linear and (b) angular properties.

Original: By linear properties, we mean that the straight-line Euclidean distance between elements is preserved, as are more powerful properties like analogies (e.g., Paris - France + Italy
Revised: By linear properties, we mean that the straight-line Euclidean distance between elements is preserved, as are more powerful properties like analogies (e.g., Paris - France + Italy

Original: ≈ Rome).
Revised: ≈ Rome).

Original: Angular properties refer to measuring angles between pairs of points (from the origin), and as a result, cosine distance is preserved.
Revised: Angular properties refer to measuring angles between pairs of points (from the origin), and as a result, cosine distance is preserved.

Original: Given a correspondence between pairs of objects across two embeddings, the classic Procrustes method, provides a closed-form solution which minimizes the sum of Euclidean distances.
Revised: Given a correspondence between pairs of objects across two embeddings, the classic Procrustes method, provides a closed-form solution which minimizes the sum of Euclidean distances.

Original: If the vectors are all first normalized, then this also maximizes the sum of cosine similarities (Dev et al., 2021).
Revised: If the vectors are all first normalized, then this also maximizes the sum of cosine similarities (Dev et al., 2021).

Original: Under this framework, there has been a flurry of work significantly improving CLWE model performance along with two directions.
Revised: Under this framework, there has been a flurry of work significantly improving CLWE model performance along with two directions.

Original: Semi-supervised and unsupervised models make these approaches require less input, and more amenable to lower-resource languages.
Revised: Semi-supervised and unsupervised models make these approaches require less input, and more amenable to lower-resource languages.

Original: For example Bootstrap Procrustes (PROC-B) (Glavaˇs et al., 2019; Vuli´c et al., 2019) is semi-supervised in that it starts with a small pairwise correspondence (of 500-1000 words), aligns those to infer a larger correspondence, and repeats applying Procrustes alignment.
Revised: For example Bootstrap Procrustes (PROC-B) (Glavaˇs et al., 2019; Vuli´c et al., 2019) is semi-supervised in that it starts with a small pairwise correspondence (of 500-1000 words), aligns those to infer a larger correspondence, and repeats applying Procrustes alignment.

Original: Methods like MUSE (Conneau et al., 2018a) are unsupervised, and use a GAN to estimate a correspondence before applying a Procrustes procedure.
Revised: Methods like MUSE (Conneau et al., 2018a) are unsupervised, and use a GAN to estimate a correspondence before applying a Procrustes procedure.

Original: The second direction is preprocessing the embeddings before applying the Procrustes alignment.
Revised: The second direction is preprocessing the embeddings before applying the Procrustes alignment.

Original: These involve methods like removing the mean, removing principal components, and vector length normalization, discussed later.
Revised: These involve methods like removing the mean, removing principal components, and vector length normalization, discussed later.

Original: In principle, these methods aim to remove the geometry of data intrinsic to particular languages (but not shared across languages) while preserving similarity properties as assured by orthogonal alignment.
Revised: In principle, these methods aim to remove the geometry of data intrinsic to particular languages (but not shared across languages) while preserving similarity properties as assured by orthogonal alignment.

Original: The space of transformations allowed under orthogonal alignments is quite large, and we make the point that unless this data geometry is “normalized” it inhibits the alignment from optimizing over the entirety of this large space.
Revised: The space of transformations allowed under orthogonal alignments is quite large, and we make the point that unless this data geometry is “normalized” it inhibits the alignment from optimizing over the entirety of this large space.

Original: Finally, we note that methods like Canonical Correlation Analysis (CCA) (Faruqui & Dyer, 2014),
Revised: Finally, we note that methods like Canonical Correlation Analysis (CCA) (Faruqui & Dyer, 2014),

Original: Discriminative Latent Variable (DLV) (Ruder et al., 2018), and Ranking-based optimization (RCSLS) (Joulin et al., 2018) have also been applied towards finding an orthogonal alignment (or pair of alignments) which minimizes a different optimization function – since the objective function may not align with sums of squared Euclidean or cosine distance (Conneau et al., 2018a; Smith et al., 2017).
Revised: Discriminative Latent Variable (DLV) (Ruder et al., 2018), and Ranking-based optimization (RCSLS) (Joulin et al., 2018) have also been applied towards finding an orthogonal alignment (or pair of alignments) which minimizes a different optimization function – since the objective function may not align with sums of squared Euclidean or cosine distance (Conneau et al., 2018a; Smith et al., 2017).

Original: Unlike the others, the RCSLS method notably does not require a rigid transformation.
Revised: Unlike the others, the RCSLS method notably does not require a rigid transformation.

Original: This paper is on embedding preprocessing , and is agnostic to the alignment used afterward.
Revised: This paper is on embedding preprocessing , and is agnostic to the alignment used afterward.

Original: Our contribution.
Revised: Our contribution.

Original: This work proposes a new and general approach to preprocessing word embeddings, subsuming many previous approaches.
Revised: This work proposes a new and general approach to preprocessing word embeddings, subsuming many previous approaches.

Original: The key is Spectral Normalization which regularizes the spectral properties of monolingual embeddings by setting all of the top singular vectors to have the same singular value.
Revised: The key is Spectral Normalization which regularizes the spectral properties of monolingual embeddings by setting all of the top singular vectors to have the same singular value.

Original: It leaves alone the smaller singular values; these capture important information and cannot be zeroed out, but making them the same value as the top singular vectors introduces too much noise.
Revised: It leaves alone the smaller singular values; these capture important information and cannot be zeroed out, but making them the same value as the top singular vectors introduces too much noise.

Original: Spectral normalization already performs as well as the best previous approaches on alignment and translation tasks, and since it applies a fairly uniform stretching to the embeddings it does not distort monolingual similarity performance.
Revised: Spectral normalization already performs as well as the best previous approaches on alignment and translation tasks, and since it applies a fairly uniform stretching to the embeddings it does not distort monolingual similarity performance.

Original: Moreover, we show layering Spectral Normalization within an iterative sequence with also centering and vector length normalization improves results further.
Revised: Moreover, we show layering Spectral Normalization within an iterative sequence with also centering and vector length normalization improves results further.

Original: We demonstrate this improvement on the standard translation tasks, as well as on downstream cross-lingual use cases of document classification and natural language inference.
Revised: We demonstrate this improvement on the standard translation tasks, as well as on downstream cross-lingual use cases of document classification and natural language inference.

Original: We provide code at https://github.com/poaboagye/SpecNorm .
Revised: We provide code at https://github.com/poaboagye/SpecNorm .

Original: Beyond CLWE.
Revised: Beyond CLWE.

Original: This paper focuses on normalization for the alignment of word vector embeddings.
Revised: This paper focuses on normalization for the alignment of word vector embeddings.

Original: This is for a couple of reasons.
Revised: This is for a couple of reasons.

Original: First, most relevant prior work does the same in the context of language translation, and so it allows for direct comparison.
Revised: First, most relevant prior work does the same in the context of language translation, and so it allows for direct comparison.

Original: Second, these contexts have generally the largest embeddings, are trained on the most data, and are easy to interpret.
Revised: Second, these contexts have generally the largest embeddings, are trained on the most data, and are easy to interpret.

Original: However, as foundation models (Bommasani et al., 2021) become ubiquitous, many other settings induce similar challenges.
Revised: However, as foundation models (Bommasani et al., 2021) become ubiquitous, many other settings induce similar challenges.

Original: For instance, alignment of various models from different data sources allows for improved ensemble embeddings (Dev et al., 2021), and this could extend to image (Frome et al., 2013;
Revised: For instance, alignment of various models from different data sources allows for improved ensemble embeddings (Dev et al., 2021), and this could extend to image (Frome et al., 2013;

Original: Kiela & Bottou, 2014) or graph embeddings (Grover & Leskovec, 2016; Perozzi et al., 2014).
Revised: Kiela & Bottou, 2014) or graph embeddings (Grover & Leskovec, 2016; Perozzi et al., 2014).

Original: Or in aligning embeddings of spatial data (Jenkins et al., 2019) or of merchants (Wang et al., 2021), one may want to align data from different geographic regions for tasks in transfer learning and domain adaptation.
Revised: Or in aligning embeddings of spatial data (Jenkins et al., 2019) or of merchants (Wang et al., 2021), one may want to align data from different geographic regions for tasks in transfer learning and domain adaptation.

Original: As an example of generality, we demonstrate an improvement via our normalization approaches in aligning genomic data embeddings (Demetci et al., 2020) in Appendix G.
Revised: As an example of generality, we demonstrate an improvement via our normalization approaches in aligning genomic data embeddings (Demetci et al., 2020) in Appendix G.

Original:  E XISTING M ETHODS FOR O RTHOGONAL V ECTOR S PACES A LIGNMENT
Revised:  E XISTING M ETHODS FOR O RTHOGONAL V ECTOR S PACES A LIGNMENT

Original: In an embedded representation of a set of n words, each i th word is associated with a vector a i ∈ R d , so
Revised: In an embedded representation of a set of n words, each i th word is associated with a vector a i ∈ R d , so

Original: A = { a 1 , . . . , a n } is the set of n word vector representation.
Revised: A = { a 1 , . . . , a n } is the set of n word vector representation.

Original: These vector representations (derived by methods like word2vec (Mikolov et al., 2013a), GloVe (Pennington et al., 2014), or
Revised: These vector representations (derived by methods like word2vec (Mikolov et al., 2013a), GloVe (Pennington et al., 2014), or

Original: FastText (Bojanowski et al., 2017)) are chosen so words with similar pairwise cosine similarity are found in the similar local context in large text corpora on which they are trained.
Revised: FastText (Bojanowski et al., 2017)) are chosen so words with similar pairwise cosine similarity are found in the similar local context in large text corpora on which they are trained.

Original: Higher-level linear structure is shown to emerge, such as concept subspaces and analogies (Mikolov et al., 2013).
Revised: Higher-level linear structure is shown to emerge, such as concept subspaces and analogies (Mikolov et al., 2013).

Original: The focus of this paper is on aligning embeddings of two languages L 1 and L 2 .
Revised: The focus of this paper is on aligning embeddings of two languages L 1 and L 2 .

Original: Each embedding
Revised: Each embedding

Original: A L 1 and A L 2 , is only designed to ensure pairwise relationships between its word vectors, but the actual coordinates of those vectors do not have any explicit meaning.
Revised: A L 1 and A L 2 , is only designed to ensure pairwise relationships between its word vectors, but the actual coordinates of those vectors do not have any explicit meaning.

Original: Yet, previous work has clearly demonstrated that there exists significant overall structural similarity, and alignment seeks to make correspondences between those structures for translation and joint understanding.
Revised: Yet, previous work has clearly demonstrated that there exists significant overall structural similarity, and alignment seeks to make correspondences between those structures for translation and joint understanding.

Original: 2. P RE - PROCESSING E MBEDDINGS BEFORE O RTHOGONAL A LIGNMENT
Revised: 2. P RE - PROCESSING E MBEDDINGS BEFORE O RTHOGONAL A LIGNMENT

Original: It turns out directly aligning embeddings from two languages (even using the “optimal” Procrustes solution) does not provide the best possible joint embedding for translation tasks.
Revised: It turns out directly aligning embeddings from two languages (even using the “optimal” Procrustes solution) does not provide the best possible joint embedding for translation tasks.

Original: While word meaning appears to hold a similar structure, languages have other properties such as differing word frequency, and this for instance leads to more frequent words having longer vectors in embeddings.
Revised: While word meaning appears to hold a similar structure, languages have other properties such as differing word frequency, and this for instance leads to more frequent words having longer vectors in embeddings.

Original: This extra language-specific structure tends to interfere with alignment.
Revised: This extra language-specific structure tends to interfere with alignment.

Original: As a result, a number of techniques have been developed to “normalize” the embeddings before Procrustes (or other) alignment.
Revised: As a result, a number of techniques have been developed to “normalize” the embeddings before Procrustes (or other) alignment.

Original: This, in some sense, allows the word meaning to dominate the optimization tasks without other confounding factors.
Revised: This, in some sense, allows the word meaning to dominate the optimization tasks without other confounding factors.

Original: We review the most common normalization approaches.
Revised: We review the most common normalization approaches.

Original: Mean Centering (C) subtracts the mean of all vectors in an embedding from each vector in that embedding.
Revised: Mean Centering (C) subtracts the mean of all vectors in an embedding from each vector in that embedding.

Original: The result is that the mean of all vectors is 0 .
Revised: The result is that the mean of all vectors is 0 .

Original: This is a rigid transformation, and so does not change the Euclidean distance between any pair of points in an embedding, and also preserves any linear property like analogies (e.g., Paris - France + Italy ≈ Rome).
Revised: This is a rigid transformation, and so does not change the Euclidean distance between any pair of points in an embedding, and also preserves any linear property like analogies (e.g., Paris - France + Italy ≈ Rome).

Original: Dev et al.
Revised: Dev et al.

Original: (2021) points out that this is the first step (followed by the Procrustes orthogonal transformation) to minimize the sum of squared Euclidean distances among paired words, under any rigid transformation.
Revised: (2021) points out that this is the first step (followed by the Procrustes orthogonal transformation) to minimize the sum of squared Euclidean distances among paired words, under any rigid transformation.

Original: However, this does change the cosine distance between pairs of points.
Revised: However, this does change the cosine distance between pairs of points.

Original: Length Normalization (L) makes each vector have a 2 -norm equal to 1 , but retains its direction from the origin (Artetxe et al., 2016;
Revised: Length Normalization (L) makes each vector have a 2 -norm equal to 1 , but retains its direction from the origin (Artetxe et al., 2016;

Original: Xing et al., 2015).
Revised: Xing et al., 2015).

Original: This preprocessing step does not change the cosine distance between any pair of points in an embedding.
Revised: This preprocessing step does not change the cosine distance between any pair of points in an embedding.

Original: But, it does change the Euclidean distance between pairs of points.
Revised: But, it does change the Euclidean distance between pairs of points.

Original: Despite these contrasting goals, these two normalizations each turn out to be individually effective in regularizing the geometry of the embeddings, and allow for better CLWE.
Revised: Despite these contrasting goals, these two normalizations each turn out to be individually effective in regularizing the geometry of the embeddings, and allow for better CLWE.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019), realized doing both was even more effective, and showed that iterating these two steps achieves the state-of-the-art way to preprocess, we denote as I-C+L .
Revised: (2019), realized doing both was even more effective, and showed that iterating these two steps achieves the state-of-the-art way to preprocess, we denote as I-C+L .

Original: Iterative Normalization eventually transforms monolingual word embeddings to have unit-length and zero-mean simultaneously (in practice they terminate this iterative process after a few steps before it achieves these two goals exactly).
Revised: Iterative Normalization eventually transforms monolingual word embeddings to have unit-length and zero-mean simultaneously (in practice they terminate this iterative process after a few steps before it achieves these two goals exactly).

Original: PCA Removal (PR) computes the principal component analysis (PCA) of an embedding, and then projects away from the direction of the top principal component, removing it.
Revised: PCA Removal (PR) computes the principal component analysis (PCA) of an embedding, and then projects away from the direction of the top principal component, removing it.

Original: Mu & Viswanath (2018) observed these directions typically do not encode essential semantic relationships between words but rather align strongly with word frequency; and PCA removal before alignment led to improved performance on several tasks.
Revised: Mu & Viswanath (2018) observed these directions typically do not encode essential semantic relationships between words but rather align strongly with word frequency; and PCA removal before alignment led to improved performance on several tasks.

Original: Sachidananda et al.
Revised: Sachidananda et al.

Original: (2021) also showed this preprocessing improved the BLI task.
Revised: (2021) also showed this preprocessing improved the BLI task.

Original: 2. S PECTRAL S TATISTICS OF E MBEDDINGS
Revised: 2. S PECTRAL S TATISTICS OF E MBEDDINGS

Original: Dubossarsky et al.
Revised: Dubossarsky et al.

Original: (2020) recently documented how cross-lingual alignment is strongly affected by the spectral statistics of monolingual embeddings.
Revised: (2020) recently documented how cross-lingual alignment is strongly affected by the spectral statistics of monolingual embeddings.

Original: We stack the embedded vectors a i ∈ R d as rows in a n × d matrix A ∈ R n × d .
Revised: We stack the embedded vectors a i ∈ R d as rows in a n × d matrix A ∈ R n × d .

Original: The SVD decomposes A into U Σ V ⊤ where U and V contain the left and right singular vectors, and the singular values σ 1 ≥ σ 2 ≥ . . .
Revised: The SVD decomposes A into U Σ V ⊤ where U and V contain the left and right singular vectors, and the singular values σ 1 ≥ σ 2 ≥ . . .

Original: ≥ σ d ≥ 0 are on the diagonal of Σ .
Revised: ≥ σ d ≥ 0 are on the diagonal of Σ .

Original: The effective rank of A is a smoother analog to rank (when there is noise in low rank components), defined er ( A )
Revised: The effective rank of A is a smoother analog to rank (when there is noise in low rank components), defined er ( A )

Original: =
Revised: =

Original: e H (Σ) where H (Σ) =
Revised: e H (Σ) where H (Σ) =

Original: − (cid:80) di
Revised: − (cid:80) di

Original: =1 ¯ σ i log ¯ σ i with ¯ σ i =
Revised: =1 ¯ σ i log ¯ σ i with ¯ σ i =

Original: σ i / (cid:80) di =1 σ i .
Revised: σ i / (cid:80) di =1 σ i .

Original: The effective condition number κ eff ( A )
Revised: The effective condition number κ eff ( A )

Original: =
Revised: =

Original: σ 1 /σ er ( A ) , which replaces the numerator (of condition number, σ d ) with the more robust singular value at the effective rank.
Revised: σ 1 /σ er ( A ) , which replaces the numerator (of condition number, σ d ) with the more robust singular value at the effective rank.

Original: This is desired to be small in stable data sets.
Revised: This is desired to be small in stable data sets.

Original: The joint effective condition number measures the harmonic mean of the effective condition number across two matrices A, A ′ as ECOND - HM
Revised: The joint effective condition number measures the harmonic mean of the effective condition number across two matrices A, A ′ as ECOND - HM

Original: ( A, A ′ )
Revised: ( A, A ′ )

Original: =
Revised: =

Original: 2 κ eff ( A ) κ eff
Revised: 2 κ eff ( A ) κ eff

Original: ( A ′ )
Revised: ( A ′ )

Original: κ eff ( A )
Revised: κ eff ( A )

Original: + κ eff ( A ′ ) .
Revised: + κ eff ( A ′ ) .

Original: The singular value gap measures how similar the singular value sequences are between two matrices as SVG ( A, A ′ )
Revised: The singular value gap measures how similar the singular value sequences are between two matrices as SVG ( A, A ′ )

Original: = (cid:80) di
Revised: = (cid:80) di

Original: =1 (log σ i − log σ ′ i ) 2 .
Revised: =1 (log σ i − log σ ′ i ) 2 .

Original: These should both be smaller, for more comparable data sets.
Revised: These should both be smaller, for more comparable data sets.

Original: Dubossarsky et al.
Revised: Dubossarsky et al.

Original: (2020) applied these to monolingual embeddings and demonstrated that the performances of several CLWE methods were closely tied to these spectral properties.
Revised: (2020) applied these to monolingual embeddings and demonstrated that the performances of several CLWE methods were closely tied to these spectral properties.

Original: Basically embeddings align better if they are better jointly conditioned, especially measured via joint effective condition number and the singular value gap.
Revised: Basically embeddings align better if they are better jointly conditioned, especially measured via joint effective condition number and the singular value gap.

Original: Contextual embeddings have also been shown to suffer from similar challenges (Ethayarajh, 2019; Xu & Koehn, 2021).
Revised: Contextual embeddings have also been shown to suffer from similar challenges (Ethayarajh, 2019; Xu & Koehn, 2021).

Original: Motivated by this idea, we propose methods that spectrally normalize embeddings improving these statistics while retaining intra-embedding meaning.
Revised: Motivated by this idea, we propose methods that spectrally normalize embeddings improving these statistics while retaining intra-embedding meaning.

Original: center + length + spectral normalized nekoinusharyo basu ichi nojo le shigoto cat dog car bus city farm homework
Revised: center + length + spectral normalized nekoinusharyo basu ichi nojo le shigoto cat dog car bus city farm homework

Original: Best Alignment center + length normalized cat dog car bus city farm homework
Revised: Best Alignment center + length normalized cat dog car bus city farm homework

Original: English nekoinusharyo
Revised: English nekoinusharyo

Original: basu ichi nojo le shigoto
Revised: basu ichi nojo le shigoto

Original: Japanese neko inu sharyo basu ichi nojo le shigoto
Revised: Japanese neko inu sharyo basu ichi nojo le shigoto

Original: Japanese cat dog car bus city farm home work
Revised: Japanese cat dog car bus city farm home work

Original: English neko inu sharyo basu ichi nojo le shigoto ca t
Revised: English neko inu sharyo basu ichi nojo le shigoto ca t

Original: d
Revised: d

Original: og ca r b u s c i t
Revised: og ca r b u s c i t

Original: y f a r m h
Revised: y f a r m h

Original: o m e w o r k
Revised: o m e w o r k

Original: Best Alignment
Revised: Best Alignment

Original:  N EW M ETHOD : S PECTRAL N ORMALIZATION
Revised:  N EW M ETHOD : S PECTRAL N ORMALIZATION

Original: Unit-length and zero-mean normalization makes embedding vectors from a language lie on a hypersphere with the center of the hypersphere centered at the origin.
Revised: Unit-length and zero-mean normalization makes embedding vectors from a language lie on a hypersphere with the center of the hypersphere centered at the origin.

Original: However, this does not take into account how the word embeddings vectors are clustered on the hypersphere.
Revised: However, this does not take into account how the word embeddings vectors are clustered on the hypersphere.

Original: See example in
Revised: See example in

Original: Figure 1 where despite centering and length normalization, words are grouped differently across languages, and this prevents a close alignment.
Revised: Figure 1 where despite centering and length normalization, words are grouped differently across languages, and this prevents a close alignment.

Original: Approaches like PCA removal and mean centering have the effect of reducing the top principal component or top singular vector.
Revised: Approaches like PCA removal and mean centering have the effect of reducing the top principal component or top singular vector.

Original: As a result, if the spectral properties are extreme, it can help regularize them.
Revised: As a result, if the spectral properties are extreme, it can help regularize them.

Original: However, this approach can be blunt.
Revised: However, this approach can be blunt.

Original: PCA removal makes the top singular value exactly 0, so the condition number becomes infinite.
Revised: PCA removal makes the top singular value exactly 0, so the condition number becomes infinite.

Original: To this effect, we propose a new algorithm Spectral Normalization that more gently regularizes the spectral properties of word embeddings; see Algorithm 1.
Revised: To this effect, we propose a new algorithm Spectral Normalization that more gently regularizes the spectral properties of word embeddings; see Algorithm 1.

Original: We will then combine it with other approaches to again ensure the embedding vectors lie on the unit sphere.
Revised: We will then combine it with other approaches to again ensure the embedding vectors lie on the unit sphere.

Original: In Figure 1, this approach spreads out words on the unit sphere without reordering them, and allows for a closer alignment.
Revised: In Figure 1, this approach spreads out words on the unit sphere without reordering them, and allows for a closer alignment.

Original: This updates part of the spectral properties of embedding A as a whole, using a parameter β ≥(fixed later as β = 2 via cross-validation).
Revised: This updates part of the spectral properties of embedding A as a whole, using a parameter β ≥(fixed later as β = 2 via cross-validation).

Original: Based on an average of singular values η = (cid:112) ∥
Revised: Based on an average of singular values η = (cid:112) ∥

Original: A ∥ 2 F /d , if a value is above β times that average, it adjusts it to βη .
Revised: A ∥ 2 F /d , if a value is above β times that average, it adjusts it to βη .

Original: Hence, all of the top directions are given the same singular value.
Revised: Hence, all of the top directions are given the same singular value.

Original: Otherwise, if it is below βη , it is considered a minor effect (some are quite small, and fairly noisy), and it is left alone.
Revised: Otherwise, if it is below βη , it is considered a minor effect (some are quite small, and fairly noisy), and it is left alone.

Original: If these small ones are completely zeroed out, the critical information within is destroyed.
Revised: If these small ones are completely zeroed out, the critical information within is destroyed.

Original: However, if these small ones are given the same value (i.e., βη ) then components which do not contribute to the most prevalent aspects of a vectors similarity are given more importance, and we observe that the usefulness of the embedding decreases.
Revised: However, if these small ones are given the same value (i.e., βη ) then components which do not contribute to the most prevalent aspects of a vectors similarity are given more importance, and we observe that the usefulness of the embedding decreases.

Original: Iterative Spectral Normalization.
Revised: Iterative Spectral Normalization.

Original: Spectral normalization makes the most sense (see Appendix
Revised: Spectral normalization makes the most sense (see Appendix

Original: J) in a setting where the vectors are already centered, and also unit length.
Revised: J) in a setting where the vectors are already centered, and also unit length.

Original: While SpecNorm does not change the center of the data, it does not maintain the length of individual vectors.
Revised: While SpecNorm does not change the center of the data, it does not maintain the length of individual vectors.

Original: As such, we advocate combining these methods into a single iterative algorithm: I-C+SN+L as in Algorithm 2.
Revised: As such, we advocate combining these methods into a single iterative algorithm: I-C+SN+L as in Algorithm 2.

Original: 1: for #Iter steps do
Revised: 1: for #Iter steps do

Original: We observe in Figure 2 that this process significantly improves the spectral properties compared to any other approach.
Revised: We observe in Figure 2 that this process significantly improves the spectral properties compared to any other approach.

Original: Without preprocessing (None), the languages (EN: English, DE: German, HI:
Revised: Without preprocessing (None), the languages (EN: English, DE: German, HI:

Original: Hindi, JA: Japanese shown) have large effective condition numbers – indicating that there is a large disparity between meaningful singular values.
Revised: Hindi, JA: Japanese shown) have large effective condition numbers – indicating that there is a large disparity between meaningful singular values.

Original: Note the y-axis is in the log scale.
Revised: Note the y-axis is in the log scale.

Original: Hence, aligning these languages without normalization would likely restrict alignment among top singular vectors, not allowing enough degree of freedom to align corresponding words.
Revised: Hence, aligning these languages without normalization would likely restrict alignment among top singular vectors, not allowing enough degree of freedom to align corresponding words.

Original: In contrast, after preprocessing when these values are more uniform, rotations among the dimensions containing the top principal components will not have an influence on the data distribution
Revised: In contrast, after preprocessing when these values are more uniform, rotations among the dimensions containing the top principal components will not have an influence on the data distribution

Original: [Figure]
Revised: [Figure]

Original: and can fully optimize the alignment between words.
Revised: and can fully optimize the alignment between words.

Original: Moreover, Figure 2 shows that I-C+SN+L most decreases the effective condition number, joint effective condition number
Revised: Moreover, Figure 2 shows that I-C+SN+L most decreases the effective condition number, joint effective condition number

Original: [Figure]
Revised: [Figure]

Original: and singular value gap.
Revised: and singular value gap.

Original: Further, these values are fairly uniform across languages, despite great variation beforehand (as shown with None).
Revised: Further, these values are fairly uniform across languages, despite great variation beforehand (as shown with None).

Original: In fact, I-C+SN+L is much more effective than other methods.
Revised: In fact, I-C+SN+L is much more effective than other methods.

Original: [Figure]
Revised: [Figure]

Original: Is Iteration Necessary?
Revised: Is Iteration Necessary?

Original: While alternating optimization is a common paradigm, is it necessary or useful to achieve these normalization goals?
Revised: While alternating optimization is a common paradigm, is it necessary or useful to achieve these normalization goals?

Original: There may be many solutions which achieve length, center
Revised: There may be many solutions which achieve length, center

Original: [Figure]
Revised: [Figure]

Original: and spectral normalization simultaneously.
Revised: and spectral normalization simultaneously.

Original: In fact, we observe that “centering” with the geometric median (Chatelon et al., 1978; Eyster et al., 1973; Overton, 1983) and then length normalizing vectors (we call this GeoMediaN , and detail it in Appendix A) achieves length and center normalization without iteration.
Revised: In fact, we observe that “centering” with the geometric median (Chatelon et al., 1978; Eyster et al., 1973; Overton, 1983) and then length normalizing vectors (we call this GeoMediaN , and detail it in Appendix A) achieves length and center normalization without iteration.

Original: However, as we observe, it under-performs I-C+L.
Revised: However, as we observe, it under-performs I-C+L.

Original:  E XPERIMENTAL A NALYSIS
Revised:  E XPERIMENTAL A NALYSIS

Original: We provide an evaluation of our proposed preprocessing methods using eight (8) language embeddings pre-trained on Wikipedia (Bojanowski et al., 2017) of each language: Croatian (HR), English (EN), Finnish (FI), French (FR), German (DE), Italian (IT), Russian (RU), and Turkish (TR).
Revised: We provide an evaluation of our proposed preprocessing methods using eight (8) language embeddings pre-trained on Wikipedia (Bojanowski et al., 2017) of each language: Croatian (HR), English (EN), Finnish (FI), French (FR), German (DE), Italian (IT), Russian (RU), and Turkish (TR).

Original: We use the 300-dimensional fastText (Bojanowski et al., 2017) embeddings, and all vocabularies are trimmed to the 200K most frequent words.
Revised: We use the 300-dimensional fastText (Bojanowski et al., 2017) embeddings, and all vocabularies are trimmed to the 200K most frequent words.

Original: Alignment evaluation tasks: BLI We evaluate and compare our proposed preprocessing methods mostly on the Bilingual Lexicon Induction (BLI) task, a word translation task.
Revised: Alignment evaluation tasks: BLI We evaluate and compare our proposed preprocessing methods mostly on the Bilingual Lexicon Induction (BLI) task, a word translation task.

Original: BLI has become the de facto evaluation task for CLWE models.
Revised: BLI has become the de facto evaluation task for CLWE models.

Original: For words in the source language, this task retrieves the nearest neighbors in the target language after alignment to check if it contains the translation.
Revised: For words in the source language, this task retrieves the nearest neighbors in the target language after alignment to check if it contains the translation.

Original: It reports the mean average precision (MAP) (Glavaˇs et al., 2019), which is equivalent to the mean reciprocal rank (MRR), of the translation.
Revised: It reports the mean average precision (MAP) (Glavaˇs et al., 2019), which is equivalent to the mean reciprocal rank (MRR), of the translation.

Original: Unless stated otherwise, reported values on baseline methods are taken from (Glavaˇs et al., 2019), and use the Google Translate (GTrans) dictionary from (Glavaˇs et al., 2019).
Revised: Unless stated otherwise, reported values on baseline methods are taken from (Glavaˇs et al., 2019), and use the Google Translate (GTrans) dictionary from (Glavaˇs et al., 2019).

Original: We trained (aligned) using 1k, 3k and 5k source words and evaluated (tested) on separate 2k source test queries, unless noted otherwise.
Revised: We trained (aligned) using 1k, 3k and 5k source words and evaluated (tested) on separate 2k source test queries, unless noted otherwise.

Original: Alignment Algorithms.
Revised: Alignment Algorithms.

Original: We evaluated and compared the result of several supervised rigidtransformation CLWE models on the evaluation benchmarks using our proposed methods.
Revised: We evaluated and compared the result of several supervised rigidtransformation CLWE models on the evaluation benchmarks using our proposed methods.

Original: All have publicly available codes, links are found in the reference citation.
Revised: All have publicly available codes, links are found in the reference citation.

Original: These include Canonical Correlation Analysis (CCA) (Faruqui & Dyer, 2014), Procrustes (PROC) (Artetxe et al., 2016; Smith et al., 2017; Xing et al., 2015; Dev et al., 2021), Bootstrapping Procrustes (PROC-B) (Glavaˇs et al., 2019), and Discriminative Latent-Variable (DLV) (Ruder et al., 2018), as discussed in Section 1.
Revised: These include Canonical Correlation Analysis (CCA) (Faruqui & Dyer, 2014), Procrustes (PROC) (Artetxe et al., 2016; Smith et al., 2017; Xing et al., 2015; Dev et al., 2021), Bootstrapping Procrustes (PROC-B) (Glavaˇs et al., 2019), and Discriminative Latent-Variable (DLV) (Ruder et al., 2018), as discussed in Section 1.

Original: We also consider Ranking-Based Optimization (RCSLS)
Revised: We also consider Ranking-Based Optimization (RCSLS)

Original: (Joulin et al., 2018) which is not a rigid alignment.
Revised: (Joulin et al., 2018) which is not a rigid alignment.

Original: In a few places, we also compare with VECMAP (Artetxe et al., 2018) as an example of an unsupervised alignment process.
Revised: In a few places, we also compare with VECMAP (Artetxe et al., 2018) as an example of an unsupervised alignment process.

Original: This should only use the geometry of the global embedding structure, e.g., derived from the natural ontology, and our normalization method still helps.
Revised: This should only use the geometry of the global embedding structure, e.g., derived from the natural ontology, and our normalization method still helps.

Original: 4. H YPERPARAMETER T UNING
Revised: 4. H YPERPARAMETER T UNING

Original: Our main proposed algorithm I-C+SN+L has a few simple parameters.
Revised: Our main proposed algorithm I-C+SN+L has a few simple parameters.

Original: To avoid overfitting, we choose these through cross-validation on English (EN) and a held-out set of five (5) languages
Revised: To avoid overfitting, we choose these through cross-validation on English (EN) and a held-out set of five (5) languages

Original: Hindi (HI), Russian (RU), Chinese (ZH), Japanese (JA), Turkish (TR).
Revised: Hindi (HI), Russian (RU), Chinese (ZH), Japanese (JA), Turkish (TR).

Original: Ten (10) Language pairs of the form EN-X and X-EN were considered.
Revised: Ten (10) Language pairs of the form EN-X and X-EN were considered.

Original: The hyperparameters β ∈ { 1 , 2 , 3 , 4 , 5 } and #Iter (number of iterations) ∈ { 1 , 2 , 3 , 4 , 5 } were fine-tuned for I-C+SN+L.
Revised: The hyperparameters β ∈ { 1 , 2 , 3 , 4 , 5 } and #Iter (number of iterations) ∈ { 1 , 2 , 3 , 4 , 5 } were fine-tuned for I-C+SN+L.

Original: We used the publicly available MUSE translation dictionary (Conneau et al., 2018a) for hyperparameter tuning.
Revised: We used the publicly available MUSE translation dictionary (Conneau et al., 2018a) for hyperparameter tuning.

Original: The Procrustes alignment algorithm was trained on 5k source words and evaluated on 1.5k source test queries.
Revised: The Procrustes alignment algorithm was trained on 5k source words and evaluated on 1.5k source test queries.

Original: We reported the mean average precision (MAP) in Table 1 for β ∈ { 1 , 2 , 3 , 4 , 5 } and with #Iter ∈ { 1 , 2 , 3 , 4 , 5 } .
Revised: We reported the mean average precision (MAP) in Table 1 for β ∈ { 1 , 2 , 3 , 4 , 5 } and with #Iter ∈ { 1 , 2 , 3 , 4 , 5 } .

Original: We observe the value of β = 2 was consistently the best threshold (although any β ≥ 2 performed similarly).
Revised: We observe the value of β = 2 was consistently the best threshold (although any β ≥ 2 performed similarly).

Original: However, the result did not change much with respect to the number of iterations.
Revised: However, the result did not change much with respect to the number of iterations.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The tie between the #Iter hyperparameter was broken using their performance on 13 English word similarity benchmarks; details in Appendix D. In Table 2 reports the average Spearman rank coefficient score on the tasks (None means no normalization).
Revised: The tie between the #Iter hyperparameter was broken using their performance on 13 English word similarity benchmarks; details in Appendix D. In Table 2 reports the average Spearman rank coefficient score on the tasks (None means no normalization).

Original: ( β, #Iter ) =
Revised: ( β, #Iter ) =

Original: (2 , 5) achieved the highest score.
Revised: (2 , 5) achieved the highest score.

Original: So hereafter, we applying I-C+SN+L with the hyperparameter ( β, #Iter )
Revised: So hereafter, we applying I-C+SN+L with the hyperparameter ( β, #Iter )

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: ( β
Revised: ( β

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) =
Revised: ) =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: β
Revised: β

Original: [Table]
Revised: [Table]

Original: Note that the proposed approach (I-C+SN+L) only increased this score for these similarity tasks, so showed no signs of distorting inherent information.
Revised: Note that the proposed approach (I-C+SN+L) only increased this score for these similarity tasks, so showed no signs of distorting inherent information.

Original: Although Spectral Normalization does not exactly preserve the linear properties or angular properties (as centering and length normalization do, one each, respectively), it does not suffer ill effects.
Revised: Although Spectral Normalization does not exactly preserve the linear properties or angular properties (as centering and length normalization do, one each, respectively), it does not suffer ill effects.

Original: We hypothesize this is because it is somewhat uniformly stretching words along with the major modes of variation, and is effectively removing information not relevant to meaning, like frequency.
Revised: We hypothesize this is because it is somewhat uniformly stretching words along with the major modes of variation, and is effectively removing information not relevant to meaning, like frequency.

Original: This benign effect is in contrast to other spectral adjustments (removing small singular values, or setting all to the same value); see Appendix D.
Revised: This benign effect is in contrast to other spectral adjustments (removing small singular values, or setting all to the same value); see Appendix D.

Original: 4. BLI P ERFORMANCE ACROSS N ORMALIZATION AND A LIGNMENT A LGORITHMS
Revised: 4. BLI P ERFORMANCE ACROSS N ORMALIZATION AND A LIGNMENT A LGORITHMS

Original: We compare and evaluate the BLI performance (MAP) of various normalization algorithms from previous works to our proposed algorithms.
Revised: We compare and evaluate the BLI performance (MAP) of various normalization algorithms from previous works to our proposed algorithms.

Original: Using the MUSE translation dictionary, we trained
Revised: Using the MUSE translation dictionary, we trained

Original: CCA, PROC, PROC-B and RCSLS on 5k source words and evaluated on 1.5k source test queries.
Revised: CCA, PROC, PROC-B and RCSLS on 5k source words and evaluated on 1.5k source test queries.

Original: The following normalization algorithms were used in the comparison analysis: PR (PCA Removal) (Mu & Viswanath, 2018), GeoMediaN (Geometric Median Normalization), C+L (Mean centering and Length normalization, 1 round),
Revised: The following normalization algorithms were used in the comparison analysis: PR (PCA Removal) (Mu & Viswanath, 2018), GeoMediaN (Geometric Median Normalization), C+L (Mean centering and Length normalization, 1 round),

Original: I-C+L
Revised: I-C+L

Original: (Iterative Mean centering and Length normalization, 5 rounds)
Revised: (Iterative Mean centering and Length normalization, 5 rounds)

Original: (Zhang et al., 2019), SN (Spectral Normalization, 1 round), C+SN+L (Mean centering, Spectral Normalization and Length normalization, 1 round), and I-C+SN+L (Iterative Mean centering, Spectral Normalization and Length normalization, 5 rounds).
Revised: (Zhang et al., 2019), SN (Spectral Normalization, 1 round), C+SN+L (Mean centering, Spectral Normalization and Length normalization, 1 round), and I-C+SN+L (Iterative Mean centering, Spectral Normalization and Length normalization, 5 rounds).

Original: Specifically, we evaluated 18 language pairs, i.e., English (EN) from/to Bulgarian (BG), Catalan (CA), Czech (CS), German (DE), Spanish (ES), Korean (KO), Thai (TH) and Chinese (ZH), separate from hyperparameter tuning.
Revised: Specifically, we evaluated 18 language pairs, i.e., English (EN) from/to Bulgarian (BG), Catalan (CA), Czech (CS), German (DE), Spanish (ES), Korean (KO), Thai (TH) and Chinese (ZH), separate from hyperparameter tuning.

Original: The average is reported in Table 3, all results are in Appendix J. For almost all algorithms
Revised: The average is reported in Table 3, all results are in Appendix J. For almost all algorithms

Original: I-C+SN+L achieves the best scores (and especially on X L − EN , often considerably better).
Revised: I-C+SN+L achieves the best scores (and especially on X L − EN , often considerably better).

Original: The only exceptions are on non-rigid RCSLS when C+SN+L (with no iteration) or just SN (without
Revised: The only exceptions are on non-rigid RCSLS when C+SN+L (with no iteration) or just SN (without

Original: C+L) performs slightly better.
Revised: C+L) performs slightly better.

Original: So, Spectral Normalization, and in particular I-C+SN+L, is shown as the best way to normalize languages before alignment.
Revised: So, Spectral Normalization, and in particular I-C+SN+L, is shown as the best way to normalize languages before alignment.

Original: [Table]
Revised: [Table]

Original: C+SN+L 0.
Revised: C+SN+L 0.

Original: I-C+SN+L 0.
Revised: I-C+SN+L 0.

Original: We also compute the average BLI MAP score across all 28 language pairs for more direct comparison to prior work (Glava ˇ s et al., 2019), summarized in Table 4 and Appendix E. All results are in
Revised: We also compute the average BLI MAP score across all 28 language pairs for more direct comparison to prior work (Glava ˇ s et al., 2019), summarized in Table 4 and Appendix E. All results are in

Original: Appendix L.
Revised: Appendix L.

Original: We compare I-C+SN+L (denoted with SN ) against no normalization on various dictionary sizes: 1k, 3k and 5k source words and evaluated on 2k source test queries.
Revised: We compare I-C+SN+L (denoted with SN ) against no normalization on various dictionary sizes: 1k, 3k and 5k source words and evaluated on 2k source test queries.

Original: In all cases, IC+SN+L consistently improves over the baseline; see Appendix E. This includes improvement over
Revised: In all cases, IC+SN+L consistently improves over the baseline; see Appendix E. This includes improvement over

Original: RCSLS which is non-rigid, so in principle could “learn” adjustments similar to our normalization in the process of alignment.
Revised: RCSLS which is non-rigid, so in principle could “learn” adjustments similar to our normalization in the process of alignment.

Original: We also tested on VECMAP, an unsupervised approach; I-C+SN+L preprocessing also improves this result from 0.375 to 0.410.
Revised: We also tested on VECMAP, an unsupervised approach; I-C+SN+L preprocessing also improves this result from 0.375 to 0.410.

Original: Dict
Revised: Dict

Original: CC
Revised: CC

Original: A
Revised: A

Original: CC
Revised: CC

Original: A S N
Revised: A S N

Original: P R O
Revised: P R O

Original: C
Revised: C

Original: P R O
Revised: P R O

Original: C S N
Revised: C S N

Original: P R O
Revised: P R O

Original: C - B
Revised: C - B

Original: P R O
Revised: P R O

Original: C - B
Revised: C - B

Original: S N
Revised: S N

Original: D L V D L V
Revised: D L V D L V

Original: S N
Revised: S N

Original: RC S
Revised: RC S

Original: L S
Revised: L S

Original: RC S
Revised: RC S

Original: L S S N 1k .
Revised: L S S N 1k .

Original: 3k .
Revised: 3k .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Normalizing contextual type-level embeddings.
Revised: Normalizing contextual type-level embeddings.

Original: Table 5 compares the impact of Iterative Normalization and Spectral Normalization on the BLI performance on aligning Cross-lingual Contextual
Revised: Table 5 compares the impact of Iterative Normalization and Spectral Normalization on the BLI performance on aligning Cross-lingual Contextual

Original: Embedding.
Revised: Embedding.

Original: We follow the implementation details from Xu & Koehn (2021) for learning representatives within BERT (see Appendix C for more details).
Revised: We follow the implementation details from Xu & Koehn (2021) for learning representatives within BERT (see Appendix C for more details).

Original: Then the learned normalization can be viewed as a composition to the functional embedding, so is compatible with downstream uses.
Revised: Then the learned normalization can be viewed as a composition to the functional embedding, so is compatible with downstream uses.

Original: Our proposed normalization algorithm, Iterative Spectral Normalization (I-C+SN+L), clearly outperforms
Revised: Our proposed normalization algorithm, Iterative Spectral Normalization (I-C+SN+L), clearly outperforms

Original: Iterative Normalization (I-C+L) on the BLI task for aligning Contextual Embeddings.
Revised: Iterative Normalization (I-C+L) on the BLI task for aligning Contextual Embeddings.

Original: This buttresses the claim made by Xu & Koehn (2021) that improving the degree of isomorphism between contextual embeddings spaces enhances the quality of learned alignment and subsequently a better performance on the BLI task.
Revised: This buttresses the claim made by Xu & Koehn (2021) that improving the degree of isomorphism between contextual embeddings spaces enhances the quality of learned alignment and subsequently a better performance on the BLI task.

Original: 4. D OWNSTREAM T ASKS
Revised: 4. D OWNSTREAM T ASKS

Original: We conclude by demonstrating that Spectral Normalization not only improves in direct translation tasks, but also captures an important global structure that generalizes from a high resource language (i.e., English, EN) to lower resource languages.
Revised: We conclude by demonstrating that Spectral Normalization not only improves in direct translation tasks, but also captures an important global structure that generalizes from a high resource language (i.e., English, EN) to lower resource languages.

Original: In both examples, a powerful classifier is trained on the EN embedding (after normalization), and then we demonstrate that after a lower resource language (e.g., German, DE) has been normalized and aligned the analysis task can be directly applied to that language.
Revised: In both examples, a powerful classifier is trained on the EN embedding (after normalization), and then we demonstrate that after a lower resource language (e.g., German, DE) has been normalized and aligned the analysis task can be directly applied to that language.

Original: In particular, adding our normalization (I-C+SN+L) dramatically improves the results over not doing that step, and typically improves on I-C+L normalization.
Revised: In particular, adding our normalization (I-C+SN+L) dramatically improves the results over not doing that step, and typically improves on I-C+L normalization.

Original: Cross-lingual Document Classification (CLDC).
Revised: Cross-lingual Document Classification (CLDC).

Original: The CLDC task builds a topic classification using a language model on a high resource language (in our case English EN) across 15 topics.
Revised: The CLDC task builds a topic classification using a language model on a high resource language (in our case English EN) across 15 topics.

Original: The
Revised: The

Original: TED CLDC corpus assembled by (Hermann & Blunsom, 2014) was used for training and evaluation.
Revised: TED CLDC corpus assembled by Hermann & Blunsom (2014) was used for training and evaluation.

Original: Following (Glava ˇ s et al., 2019) a simple CNN was used to train.
Revised: Following Glava ˇ s et al. (2019), a simple CNN was used to train.

Original: Table 6 summarizes the average
Revised: Table 6 summarizes the average

Original: F1-score for all topic classifiers on 5 language pairs.
Revised: F1-score for all topic classifiers on 5 language pairs.

Original: The CLWEs induced by PROC SN , PROC-B SN ,
Revised: The CLWEs induced by PROC SN , PROC-B SN ,

Original: DLV SN , and RCSLS SN (using I-C+SN+L) outperformed the baseline result (with no normalization) on the CLDC task, greatly improving the best average score from 0 .
Revised: DLV SN , and RCSLS SN (using I-C+SN+L) outperformed the baseline result (with no normalization) on the CLDC task, greatly improving the best average score from 0 .

Original: 421 to 0 .
Revised: 421 to 0 .

Original: 461 , and improving over I-C+SN+L for all alignments except RCSLS.
Revised: 461 , and improving over I-C+SN+L for all alignments except RCSLS.

Original: Glavaˇs et al.
Revised: Glavaˇs et al.

Original: (2019) used only 12 of 15 topics, but could not confirm which, so we re-ran all baselines using all 15 topics.[Table]
Revised: (2019) used only 12 of 15 topics, but could not confirm which, so we re-ran all baselines using all 15 topics.

Original: [Table]
Revised: [Table]

Original: Cross-lingual Natural Language Inference (XNLI).
Revised: Cross-lingual Natural Language Inference (XNLI).

Original: We evaluated the CLWE on a cross-lingual natural language inference (XNLI) task.
Revised: We evaluated the CLWE on a cross-lingual natural language inference (XNLI) task.

Original: We used a multi-lingual XNLI corpus created by (Conneau et al., 2018b), which is a collection of sentence pairs from the English MultiNLI corpus (Williams et al., 2018) translated into 14 languages.
Revised: We used a multi-lingual XNLI corpus created by Conneau et al. (2018b), which is a collection of sentence pairs from the English MultiNLI corpus (Williams et al., 2018) translated into 14 languages.

Original: The MultiNLI corpus contains 433k sentence pairs with the labels entailment, contradiction, and neutral.
Revised: The MultiNLI corpus contains 433k sentence pairs with the labels entailment, contradiction, and neutral.

Original: The intersection between XNLI languages and BLI languages results in four XNLI evaluation pairs: EN-DE, EN-FR, EN-TR, and EN-RU.
Revised: The intersection between XNLI languages and BLI languages results in four XNLI evaluation pairs: EN-DE, EN-FR, EN-TR, and EN-RU.

Original: We use the training setup in (Glavaˇs et al., 2019) with the Enhanced Sequential Inference Model (Chen et al., 2017) on English after normalization.
Revised: We use the training setup in Glavaˇs et al. with the Enhanced Sequential Inference Model (Chen et al., 2017) on English after normalization.

Original: First, we aligned normalized versions of each language onto the normalized EN embedding to obtain the shared cross-lingual embedding.
Revised: First, we aligned normalized versions of each language onto the normalized EN embedding to obtain the shared cross-lingual embedding.

Original: Then we used the 5k test pairs from the XNLI corpus to evaluate each language alignment.
Revised: Then we used the 5k test pairs from the XNLI corpus to evaluate each language alignment.

Original: Table 7 shows the result for PROC, PROC-B, and RCSLS alignments (DLV and VECMAP transform the EN embedding in the process, so were omitted).
Revised: Table 7 shows the result for PROC, PROC-B, and RCSLS alignments (DLV and VECMAP transform the EN embedding in the process, so were omitted).

Original: We compare I-C+L and our I-C+SN+L normalization against the same procedure without normalization from Glavaˇs et al.
Revised: We compare I-C+L and our I-C+SN+L normalization against the same procedure without normalization from Glavaˇs et al.

Original: As in other experiments, our normalization improves the average test accuracy with each alignment approach.
Revised: As in other experiments, our normalization improves the average test accuracy with each alignment approach.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION & D ISCUSSION
Revised:  C ONCLUSION & D ISCUSSION

Original: We introduce a new way to normalize embeddings, based on spectral normalization, for use in creating cross-lingual word embeddings.
Revised: We introduce a new way to normalize embeddings, based on spectral normalization, for use in creating cross-lingual word embeddings.

Original: Our approach generalizes previous approaches, and when used to individually preprocess monolingual embeddings, it allows alignment procedures to find better alignments: resulting in improved performance on direct translation tasks as well as crosslingual topic classification and natural language inference tasks.
Revised: Our approach generalizes previous approaches, and when used to individually preprocess monolingual embeddings, it allows alignment procedures to find better alignments: resulting in improved performance on direct translation tasks as well as crosslingual topic classification and natural language inference tasks.

Original: Moreover, we demonstrate this improvement is very broadly useful; it holds in contextual embeddings as well as on embeddings of non-language data (on genomic data in Appendix G).
Revised: Moreover, we demonstrate this improvement is very broadly useful; it holds in contextual embeddings as well as on embeddings of non-language data (on genomic data in Appendix G).

Original: Limitations.
Revised: Limitations.

Original: Our proposed preprocessing method heavily relies on the approximately isomorphic assumption of the two monolingual embedding spaces to be aligned (Mikolov et al., 2013b; Ruder et al., 2019).
Revised: Our proposed preprocessing method heavily relies on the approximately isomorphic assumption of the two monolingual embedding spaces to be aligned (Mikolov et al., 2013b; Ruder et al., 2019).

Original: Under this condition, it is easier to learn a robust linear map between two monolingual embedding spaces that are approximately isomorphic than spaces that are not.
Revised: Under this condition, it is easier to learn a robust linear map between two monolingual embedding spaces that are approximately isomorphic than spaces that are not.

Original: Recent, lines of work (Søgaard et al., 2018; Nakashole & Flauger, 2018; Patra et al., 2018) have questioned this approximately isomorphic assumption and have shown that it does not hold in general.
Revised: Recent, lines of work (Søgaard et al., 2018; Nakashole & Flauger, 2018; Patra et al., 2018) have questioned this approximately isomorphic assumption and have shown that it does not hold in general.

Original: Also, it hinders the performance of some CLWE methods.
Revised: Also, it hinders the performance of some CLWE methods.

Original: The goal of our proposed preprocessing method is to canonically preprocess the two monolingual embedding spaces to improve their isomorphism, and we show it is generally effective.
Revised: The goal of our proposed preprocessing method is to canonically preprocess the two monolingual embedding spaces to improve their isomorphism, and we show it is generally effective.

Original: CLWE methods such as Joint CLWE methods (Wang et al., 2019) and Cross-lingual Anchoring (Ormazabal et al., 2021)
Revised: CLWE methods such as Joint CLWE methods (Wang et al., 2019) and Cross-lingual Anchoring (Ormazabal et al., 2021)

Original: departs from the approximately isomorphic assumption.
Revised: departs from the approximately isomorphic assumption.

Original: They learn new embeddings from scratch, using a different objective function that encourages the alignment of representations or the vector embeddings of the words.
Revised: They learn new embeddings from scratch, using a different objective function that encourages the alignment of representations or the vector embeddings of the words.

Original: As such our proposed preprocessing method does not apply.
Revised: As such our proposed preprocessing method does not apply.

Original: These methods, however, are less flexible and portable than alignment-based approaches, since they need to learn a new embedding for each pair of languages, and would need to re-train if only one is updated or a different or multiple pairs are considered.
Revised: These methods, however, are less flexible and portable than alignment-based approaches, since they need to learn a new embedding for each pair of languages, and would need to re-train if only one is updated or a different or multiple pairs are considered.

Original: We have a preliminary comparison with Ormazabal et al.
Revised: We have a preliminary comparison with Ormazabal et al.

Original: (2021) in Appendix F and show our methods are competitive.
Revised: (2021) in Appendix F and show our methods are competitive.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: The vast majority of NLP research and cutting-edge advancements are in English.
Revised: The vast majority of NLP research and cutting-edge advancements are in English.

Original: This disadvantages those who primarily operate in other languages, with less developed models, or less data to train models.
Revised: This disadvantages those who primarily operate in other languages, with less developed models, or less data to train models.

Original: As large language models are the cornerstone of most NLP research and development in English, one of this work’s main goals is to inexpensively port these advances to other languages, and those who use them.
Revised: As large language models are the cornerstone of most NLP research and development in English, one of this work’s main goals is to inexpensively port these advances to other languages, and those who use them.

Original: This will help unlock this technology to many around the world.
Revised: This will help unlock this technology to many around the world.

Original: As with most models, this accuracy and improvement may vary across tasks and languages.
Revised: As with most models, this accuracy and improvement may vary across tasks and languages.

Original: While language models have many positive use cases including improving accessibility, better recommendations, and increased automation, they have some negative effects as well.
Revised: While language models have many positive use cases including improving accessibility, better recommendations, and increased automation, they have some negative effects as well.

Original: These include requiring potentially large computational and hence environmental costs, encoding and exacerbating bias, and aiding in automatically generating fake or deceitful content.
Revised: These include requiring potentially large computational and hence environmental costs, encoding and exacerbating bias, and aiding in automatically generating fake or deceitful content.

Original: While this paper is unlikely to change the desire to use embeddings, it aims to reduce the burden of use and increase the effectiveness in lower-resource settings.
Revised: While this paper is unlikely to change the desire to use embeddings, it aims to reduce the burden of use and increase the effectiveness in lower-resource settings.

Original: And in particular to port models trained in English to other languages.
Revised: And in particular to port models trained in English to other languages.

Original: This would reduce the cost of retraining in other languages if the English model can be reused, easing environmental costs.
Revised: This would reduce the cost of retraining in other languages if the English model can be reused, easing environmental costs.

Original: We support the maturing efforts in attenuating bias in all such embeddings.
Revised: We support the maturing efforts in attenuating bias in all such embeddings.

Original: And while we acknowledge the possibility of this work aiding in the creation of deceitful content and the harm it can cause, we believe the many benefits outweigh the harms.
Revised: And while we acknowledge the possibility of this work aiding in the creation of deceitful content and the harm it can cause, we believe the many benefits outweigh the harms.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: All existing methods are compared with publicly available codes with publicly available data, with links above or in references.
Revised: All existing methods are compared with publicly available codes with publicly available data, with links above or in references.

Original: The exception is code for CLDC and XNLI is shared by Glavaˇs et al.
Revised: The exception is code for CLDC and XNLI is shared by Glavaˇs et al.

Original: Everything was run with default parameters; the exception is RCSLS, where we follow the suggested hyperparameter selection strategy (Joulin et al., 2018) (with learning rate in { 1 , 10 , 25 , 50 } and epoch number in { 10 , 20 } ).
Revised: Everything was run with default parameters; the exception is RCSLS, where we follow the suggested hyperparameter selection strategy (Joulin et al., 2018) (with learning rate in { 1 , 10 , 25 , 50 } and epoch number in { 10 , 20 } ).

Original: Our new code for SpecNorm is in Appendix H and here https://github.com/poaboagye/SpecNorm .
Revised: Our new code for SpecNorm is in Appendix H and here https://github.com/poaboagye/SpecNorm .

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We thank our support from NSF IIS-1816149, CCF-2115677, and Visa Research.
Revised: We thank our support from NSF IIS-1816149, CCF-2115677, and Visa Research.

Original: A PPENDIX A G EOMETRIC M EDIAN N ORMALIZATION
Revised: A PPENDIX A G EOMETRIC M EDIAN N ORMALIZATION

Original: Iterative Normalization converges towards individual word embeddings having a unit length and each monolingual embedding having a zero mean.
Revised: Iterative Normalization converges towards individual word embeddings having a unit length and each monolingual embedding having a zero mean.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019) showed that iterating solutions for these distinct goals will eventually converge to a solution which satisfies both.
Revised: (2019) showed that iterating solutions for these distinct goals will eventually converge to a solution which satisfies both.

Original: In this paper, we observe that both goals can be done in one shot without iterating – by solving the Fermat-Weber problem (Maxwell, 1966; Love et al., 1988).
Revised: In this paper, we observe that both goals can be done in one shot without iterating – by solving the Fermat-Weber problem (Maxwell, 1966; Love et al., 1988).

Original: This dates to the 17th century, and corresponds with identifying the geometric median of a point set.
Revised: This dates to the 17th century, and corresponds with identifying the geometric median of a point set.

Original: Formally, the goal is a point x ∗ ∈ R d that minimizes the sum of distances from n anchor points { a 1 , . . .
Revised: Formally, the goal is a point x ∗ ∈ R d that minimizes the sum of distances from n anchor points { a 1 , . . .

Original: , a n } ⊂ R d which are not colinear:
Revised: , a n } ⊂ R d which are not colinear:

Original: [Equation]
Revised: [Equation]

Original: Several methods (Chatelon et al., 1978; Eyster et al., 1973; Overton, 1983) have been proposed; the most popular is the Weiszfeld’s algorithm ( Weiszfeld , see below for details).
Revised: Several methods (Chatelon et al., 1978; Eyster et al., 1973; Overton, 1983) have been proposed; the most popular is the Weiszfeld’s algorithm ( Weiszfeld , see below for details).

Original: It is folklore that the solution x ∗ satisfies that 

Revised: It is folklore that the solution x ∗ satisfies that

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =

Revised: =

Original: a i − x ∗ ∥ a i − x ∗ ∥ ; we do not know of a written proof, so we prove this for completeness.
Revised: a i − x ∗ ∥ a i − x ∗ ∥ ; we do not know of a written proof, so we prove this for completeness.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: If x ∈ R d is distinct from all the given anchor points, a i , then x is the geometric median is true if and only if ( ⇐⇒ )
Revised: If x ∈ R d is distinct from all the given anchor points, a i , then x is the geometric median is true if and only if ( ⇐⇒ )

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: ( ⇒ )
Revised: ( ⇒ )

Original: Suppose x ∈ R d is distinct from all the given anchor points, a i and x is the geometric median such that x = (cid:101) T ( x ) (see A.1) then
Revised: Suppose x ∈ R d is distinct from all the given anchor points, a i and x is the geometric median such that x = (cid:101) T ( x ) (see A.1) then

Original: [Equation]
Revised: [Equation]

Original: ( ⇐ ) Suppose x ∈ R d is distinct from all the given anchor points, a i and x is the unique optimal solution of equation 1 such that ∇ f ( x ) = 0 then solving for x while ignoring the dependency x in (cid:13)(cid:13)
Revised: ( ⇐ ) Suppose x ∈ R d is distinct from all the given anchor points, a i and x is the unique optimal solution of equation 1 such that ∇ f ( x ) = 0 then solving for x while ignoring the dependency x in (cid:13)(cid:13)

Original: a i − x (cid:13)(cid:13) yields:
Revised: a i − x (cid:13)(cid:13) yields:

Original: [Equation]
Revised: [Equation]

Original: which is the geometric median.
Revised: which is the geometric median.

Original: Using this characteristic of the geometric median, we can simultaneously enforce monolingual word embeddings to have unit-length and zero-mean in just one step.
Revised: Using this characteristic of the geometric median, we can simultaneously enforce monolingual word embeddings to have unit-length and zero-mean in just one step.

Original: This can be done using the Geometric Median normalization ( GeoMediaN ) algorithm (as Algorithm 3).
Revised: This can be done using the Geometric Median normalization ( GeoMediaN ) algorithm (as Algorithm 3).

Original: Given a monolingual word embedding A , we compute the geometric median x ∗ , and ”center” the data on this point, and unit length normalizes the centered embedding.
Revised: Given a monolingual word embedding A , we compute the geometric median x ∗ , and ”center” the data on this point, and unit length normalizes the centered embedding.

Original: After these steps, all vectors are unit length, and because of the folklore property (Theorem A.1), the mean of those points is also 0 .
Revised: After these steps, all vectors are unit length, and because of the folklore property (Theorem A.1), the mean of those points is also 0 .

Original: As a result, we can state the following property.
Revised: As a result, we can state the following property.

Original: Theorem A.2.
Revised: Theorem A.2.

Original: The output of GeoMediaN
Revised: The output of GeoMediaN

Original: ( A ) is centered and length normalized.
Revised: ( A ) is centered and length normalized.

Original: Despite the Geometric Median Normalization algorithm’s ability to enforce unit-length and zeromean in just one step, we observe that it does not perform especially well on the BLI task.
Revised: Despite the Geometric Median Normalization algorithm’s ability to enforce unit-length and zeromean in just one step, we observe that it does not perform especially well on the BLI task.

Original: Both
Revised: Both

Original: GeoMediaN and I-C+L achieve one of many solutions which achieve these joint goals.
Revised: GeoMediaN and I-C+L achieve one of many solutions which achieve these joint goals.

Original: A. W EISZFELD A LGORITHM
Revised: A. W EISZFELD A LGORITHM

Original: The Weiszfeld algorithm is an iterative method for finding the geometric median of a set of points in Euclidean space based on the reformulation of a stationary point that satisfies ∇ f ( x ) = 0 .
Revised: The Weiszfeld algorithm is an iterative method for finding the geometric median of a set of points in Euclidean space based on the reformulation of a stationary point that satisfies ∇ f ( x ) = 0 .

Original: If iteration function T : R d → R d is defined by:
Revised: If iteration function T : R d → R d is defined by:

Original: [Equation]
Revised: [Equation]

Original: then the Weiszfeld algorithm is:
Revised: then the Weiszfeld algorithm is:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where x 0 ∈ R d is a starting point.
Revised: where x 0 ∈ R d is a starting point.

Original: When the current iterate, x k / ∈ (cid:8) a 1 , . . .
Revised: When the current iterate, x k / ∈ (cid:8) a 1 , . . .

Original: a n (cid:9)
Revised: a n (cid:9)

Original: [Equation]
Revised: [Equation]

Original: (cid:0) x k (cid:1) =
Revised: (cid:0) x k (cid:1) =

Original: (cid:101) T (cid:0) x k (cid:1) ;
Revised: (cid:101) T (cid:0) x k (cid:1) ;

Original: else, if x k = a i , then T (cid:0) x k (cid:1) =
Revised: else, if x k = a i , then T (cid:0) x k (cid:1) =

Original: a i .
Revised: a i .

Original: The Weiszfeld algorithm is presented in Algorithm 4 below:
Revised: The Weiszfeld algorithm is presented in Algorithm 4 below:

Original: Input
Revised: Input

Original: 4: if (cid:13)(cid:13) x k +1 − x k (cid:13)(cid:13) 2 <
Revised: 4: if (cid:13)(cid:13) x k +1 − x k (cid:13)(cid:13) 2 <

Original: ϵ then 5: return x k + 6: k ← k +
Revised: ϵ then 5: return x k + 6: k ← k +

Original: B S PECTRAL S TATISTICS AND S PECTRAL I SOMORPHISM M EASURES
Revised: B S PECTRAL S TATISTICS AND S PECTRAL I SOMORPHISM M EASURES

Original: We also explored other spectral statistics on monolingual embeddings.
Revised: We also explored other spectral statistics on monolingual embeddings.

Original: The numeric rank of A is a smoother analog to rank (where there is noise in low rank components), defined η ( A ) =
Revised: The numeric rank of A is a smoother analog to rank (where there is noise in low rank components), defined η ( A ) =

Original: ∥
Revised: ∥

Original: A ∥ 2 F / ∥
Revised: A ∥ 2 F / ∥

Original: A ∥ 22 .
Revised: A ∥ 22 .

Original: The condition number of A is κ ( A ) =
Revised: The condition number of A is κ ( A ) =

Original: σ 1 /σ d , and measures how close the matrix is to being truly full rank, smaller is more stable.
Revised: σ 1 /σ d , and measures how close the matrix is to being truly full rank, smaller is more stable.

Original: For two matrices A 

Revised: For two matrices A

Original: and A 2 , the condition number harmonic mean is COND - HM
Revised: and A 2 , the condition number harmonic mean is COND - HM

Original: ( A 1 , A 2 ) = 2 κ ( A 1 ) κ ( A 2 ) κ ( A 1 )+ κ ( A 2 ) .
Revised: ( A 1 , A 2 ) = 2 κ ( A 1 ) κ ( A 2 ) κ ( A 1 )+ κ ( A 2 ) .

Original: Smaller means the matrices are more comparable.
Revised: Smaller means the matrices are more comparable.

Original: Figure 3 plots these measures, and again demonstrates that I-C+SN+L improves these measures on matrices.
Revised: Figure 3 plots these measures, and again demonstrates that I-C+SN+L improves these measures on matrices.

Original: We also show the raw numbers used to generate the charts in Figure 2 in the tables below.
Revised: We also show the raw numbers used to generate the charts in Figure 2 in the tables below.

Original: EN
Revised: EN

Original: DE
Revised: DE

Original: HI
Revised: HI

Original: JA Language
Revised: JA Language

Original: C ond iti on N u m b e r( l og )
Revised: C ond iti on N u m b e r( l og )

Original: Condition Number
Revised: Condition Number

Original: None I-C+SN+L PR
Revised: None I-C+SN+L PR

Original: C+L I-C+L
Revised: C+L I-C+L

Original: GeoMediaN
Revised: GeoMediaN

Original: EN
Revised: EN

Original: DE
Revised: DE

Original: HI
Revised: HI

Original: JA Language
Revised: JA Language

Original: N u m e r
Revised: N u m e r

Original: i c
Revised: i c

Original: R a nk ( l og )
Revised: R a nk ( l og )

Original: Numeric Rank
Revised: Numeric Rank

Original: EN-DE
Revised: EN-DE

Original: EN-HI
Revised: EN-HI

Original: EN-JA LanguagePairs
Revised: EN-JA LanguagePairs

Original: C OND - H M ( l og )
Revised: C OND - H M ( l og )

Original: Joint Condition Number
Revised: Joint Condition Number

Original: [Table]
Revised: [Table]

Original: N on e
Revised: N on e

Original: P R
Revised: P R

Original: G e o
Revised: G e o

Original: M e d i a N
Revised: M e d i a N

Original: C + L
Revised: C + L

Original: I- C + L
Revised: I- C + L

Original: I- C + S N + L
Revised: I- C + S N + L

Original: EN
Revised: EN

Original: DE
Revised: DE

Original: HI
Revised: HI

Original: JA
Revised: JA

Original: [Table]
Revised: [Table]

Original: N on e
Revised: N on e

Original: P R
Revised: P R

Original: G e o
Revised: G e o

Original: M e d i a N
Revised: M e d i a N

Original: C + L
Revised: C + L

Original: I- C + L
Revised: I- C + L

Original: I- C + S N + L
Revised: I- C + S N + L

Original: EN 13.
Revised: EN 13.

Original: DE 14.
Revised: DE 14.

Original: HI 16.
Revised: HI 16.

Original: JA 175.
Revised: JA 175.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C N ORMALIZING CONTEXTUAL TYPE - LEVEL EMBEDDINGS
Revised: C N ORMALIZING CONTEXTUAL TYPE - LEVEL EMBEDDINGS

Original: Contextual Type-level Embeddings To obtain the contextual type-level embeddings, Fast Align (Dyer et al., 2013) is applied to the source-target parallel corpora to derive silver aligned token pairs.
Revised: Contextual Type-level Embeddings To obtain the contextual type-level embeddings, Fast Align (Dyer et al., 2013) is applied to the source-target parallel corpora to derive silver aligned token pairs.

Original: The tokenized parallel corpus is fed into pre-trained BERTs (Wolf et al., 2019; Safaya et al., 2020) of the source and target language.
Revised: The tokenized parallel corpus is fed into pre-trained BERTs (Wolf et al., 2019; Safaya et al., 2020) of the source and target language.

Original: Since there will be multiple occurrences of type-level words and each type-level word will possess a contextual word embedding, for each type-level word, its vector representation is derived from the mean vector of all the type-level words from the monolingual corpus fed into the pre-trained language model.
Revised: Since there will be multiple occurrences of type-level words and each type-level word will possess a contextual word embedding, for each type-level word, its vector representation is derived from the mean vector of all the type-level words from the monolingual corpus fed into the pre-trained language model.

Original: This is done for both source and target language.
Revised: This is done for both source and target language.

Original: The resulting type-level contextual embedding from the source and target language is then aligned by solving the Procrustes problem.
Revised: The resulting type-level contextual embedding from the source and target language is then aligned by solving the Procrustes problem.

Original: D W ORD S IMILARITY T ASK
Revised: D W ORD S IMILARITY T ASK

Original: The word similiarity task was conducted using the following English word similarity benchmarks:
Revised: The word similiarity task was conducted using the following English word similarity benchmarks:

Original: WS-3533 (Finkelstein et al., 2001), WS-SIM and WSREL (Agirre et al., 2009), RG-65 (Rubenstein & Goodenough, 1965), MC-30 (Miller & Charles, 1991), MTurk-2875 (Radinsky et al., 2011),
Revised: WS-3533 (Finkelstein et al., 2001), WS-SIM and WSREL (Agirre et al., 2009), RG-65 (Rubenstein & Goodenough, 1965), MC-30 (Miller & Charles, 1991), MTurk-2875 (Radinsky et al., 2011),

Original: MTurk-771 (Halawi et al., 2012), MEN7 (Bruni et al., 2012), YP-130 (Yang & Powers, 2006), Rare
Revised: MTurk-771 (Halawi et al., 2012), MEN7 (Bruni et al., 2012), YP-130 (Yang & Powers, 2006), Rare

Original: Words (Luong et al., 2013).
Revised: Words (Luong et al., 2013).

Original: In addition to a baseline (None, which means no normalization), Tabel 12 shows results comparing against our proposed normalization (I-C+SN+L) and the state of the art (I-C+L).
Revised: In addition to a baseline (None, which means no normalization), Tabel 12 shows results comparing against our proposed normalization (I-C+SN+L) and the state of the art (I-C+L).

Original: Note that both of these improve the accuracy of the similarity tests.
Revised: Note that both of these improve the accuracy of the similarity tests.

Original: This indicates that they are not distorting the critical information contained in the original embeddings.
Revised: This indicates that they are not distorting the critical information contained in the original embeddings.

Original: And our proposed approach (I-C+SN+L) increases the score the most.
Revised: And our proposed approach (I-C+SN+L) increases the score the most.

Original: In contrast, we show the results of two other spectral adjustments we considered.
Revised: In contrast, we show the results of two other spectral adjustments we considered.

Original: SSV (SSV means
Revised: SSV (SSV means

Original: Same Singular Values), performs the SVD on the original embedding and then sets all the singular values to η = (cid:112)
Revised: Same Singular Values), performs the SVD on the original embedding and then sets all the singular values to η = (cid:112)

Original: ∥
Revised: ∥

Original: A ∥ 2 F /d .
Revised: A ∥ 2 F /d .

Original: Then we compute U@S@VT to get the new word embedding.
Revised: Then we compute U@S@VT to get the new word embedding.

Original: So similar to SpecNorm,
Revised: So similar to SpecNorm,

Original: but it never reaches the else condition.
Revised: but it never reaches the else condition.

Original: This slightly decreases the scores on the similarity tests.
Revised: This slightly decreases the scores on the similarity tests.

Original: This is a result of accentuating the noise directions.
Revised: This is a result of accentuating the noise directions.

Original: In the other direction, SSVZ
Revised: In the other direction, SSVZ

Original: (SSVZ means Set Singular Values to Zero) only keeps the top 40 singular values and then sets the rest to zero.
Revised: (SSVZ means Set Singular Values to Zero) only keeps the top 40 singular values and then sets the rest to zero.

Original: Then we compute U@S@VT to get the new word embedding.
Revised: Then we compute U@S@VT to get the new word embedding.

Original: This drastically reduces the similarity score.
Revised: This drastically reduces the similarity score.

Original: This shows those noise directions, which tend to be well below the average singular value, are still important and cannot just be removed.
Revised: This shows those noise directions, which tend to be well below the average singular value, are still important and cannot just be removed.

Original: [Table]
Revised: [Table]

Original: E S UMMARY OF M ODEL P ERFORMANCE AND S IGNIFICANCE T EST
Revised: E S UMMARY OF M ODEL P ERFORMANCE AND S IGNIFICANCE T EST

Original: [Table]
Revised: [Table]

Original: Dict
Revised: Dict

Original: CC
Revised: CC

Original: [Table]
Revised: [Table]

Original: RC
Revised: RC

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: P 1k 28/ 28/ 25/ 28/ 13 /3k 28/ 28/ 25/ 28/ 25/ 5k 28/ 28/ 28/ 28/ – 27/
Revised: P 1k 28/ 28/ 25/ 28/ 13 /3k 28/ 28/ 25/ 28/ 25/ 5k 28/ 28/ 28/ 28/ – 27/

Original: Table 13 summarizes the performance of I-C+SN+L on several supervised and unsupervised projection-based CLWE models across all the 28 language pairs as presented in Appendix L .
Revised: Table 13 summarizes the performance of I-C+SN+L on several supervised and unsupervised projection-based CLWE models across all the 28 language pairs as presented in Appendix L .

Original: After preprocessing the monolingual word embeddings with I-C+SN+L, CCA SN , PROC SN and DLV SN outperformed CCA, PROC and DLV respectively on 28 of 28 language pairs across all the translation dictionaries.
Revised: After preprocessing the monolingual word embeddings with I-C+SN+L, CCA SN , PROC SN and DLV SN outperformed CCA, PROC and DLV respectively on 28 of 28 language pairs across all the translation dictionaries.

Original: PROC-B SN outperformed PROC-B on 25 of 28 language pairs across 1k and 3k translation dictionaries.
Revised: PROC-B SN outperformed PROC-B on 25 of 28 language pairs across 1k and 3k translation dictionaries.

Original: The performance of RCSLS SN supersedes RCSLS on 25 of 28 language pairs and 28 of 28 language pairs trained on 3k and 5k translation dictionaries respectively.
Revised: The performance of RCSLS SN supersedes RCSLS on 25 of 28 language pairs and 28 of 28 language pairs trained on 3k and 5k translation dictionaries respectively.

Original: The unsupervised projection-based CLWE model, VECMAP SN outperformed VECMAP on 27 oflanguage pairs.
Revised: The unsupervised projection-based CLWE model, VECMAP SN outperformed VECMAP on 27 oflanguage pairs.

Original: The lowest performing model was RCSLS SN trained on 1k translation dictionary.
Revised: The lowest performing model was RCSLS SN trained on 1k translation dictionary.

Original: We compare the effectiveness of Iterative Spectral Normalization on n = 28 language pairs.
Revised: We compare the effectiveness of Iterative Spectral Normalization on n = 28 language pairs.

Original: We use a 1-tail Binomial Test to measure the significance of the consistency of the improvement.
Revised: We use a 1-tail Binomial Test to measure the significance of the consistency of the improvement.

Original: That is,
Revised: That is,

Original: Iterative Spectral Normalization is a significant improvement over no normalization if it improves the score on well more than 50% of the language pairs.
Revised: Iterative Spectral Normalization is a significant improvement over no normalization if it improves the score on well more than 50% of the language pairs.

Original: Table 14 shows the p -value derived from the
Revised: Table 14 shows the p -value derived from the

Original: Binomial test with n = 28 language with probability 0 .
Revised: Binomial test with n = 28 language with probability 0 .

Original: 5 , for different values k/n language pairs with an improvement.
Revised: 5 , for different values k/n language pairs with an improvement.

Original: For instance, 21 / 28 language pairs with an improvement have a p -value of 0 .
Revised: For instance, 21 / 28 language pairs with an improvement have a p -value of 0 .

Original: 006 so is significant at the 0 .
Revised: 006 so is significant at the 0 .

Original: 05 level and the 0 .
Revised: 05 level and the 0 .

Original: 01 level.
Revised: 01 level.

Original: In the associated Tables 13 and 15 we show those results in bold which are significant at the 0 .
Revised: In the associated Tables 13 and 15 we show those results in bold which are significant at the 0 .

Original: 01 level.
Revised: 01 level.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 15 summarizes the performance of I-C+L and I-C+SN+L on several supervised and unsupervised projection-based CLWE models across all the 28 language pairs as presented in Appendix K .
Revised: Table 15 summarizes the performance of I-C+L and I-C+SN+L on several supervised and unsupervised projection-based CLWE models across all the 28 language pairs as presented in Appendix K .

Original: In all cases, I-C+SN+L outperforms I-C+L except on RCSLS where there was a tie.
Revised: In all cases, I-C+SN+L outperforms I-C+L except on RCSLS where there was a tie.

Original: Bold signifies that they are significant at a significance level of 0.01 under a Binomial Test, as shown in Table 14.
Revised: Bold signifies that they are significant at a significance level of 0.01 under a Binomial Test, as shown in Table 14.

Original: F P@1 P ERFORMANCE FOR M APPING BASED CLWE METHODS VS J OINT T RAINING
Revised: F P@1 P ERFORMANCE FOR M APPING BASED CLWE METHODS VS J OINT T RAINING

Original: [Table]
Revised: [Table]

Original: None
Revised: None

Original: [Table] [Table]
Revised: [Table] [Table]

Original: None
Revised: None

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Conneau
Revised: Conneau

Original: [Table]
Revised: [Table]

Original: al., 2018a
Revised: al., 2018a

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: None
Revised: None

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: None
Revised: None

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (Ormazabal et al., 2021)
Revised: (Ormazabal et al., 2021)

Original: [Table]
Revised: [Table]

Original: None
Revised: None

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Joint training methods.
Revised: Joint training methods.

Original: The CLWE methods proposed by Wang et al.
Revised: The CLWE methods proposed by Wang et al.

Original: (2019) and Ormazabal et al.
Revised: (2019) and Ormazabal et al.

Original: (2021) learn a new embedding from scratch including words from both languages.
Revised: (2021) learn a new embedding from scratch including words from both languages.

Original: They do so with a different objective function than standard approaches so that it prevents words entirely clustering by language, and so cross-lingual synonyms are encouraged to be embedded close to each other.
Revised: They do so with a different objective function than standard approaches so that it prevents words entirely clustering by language, and so cross-lingual synonyms are encouraged to be embedded close to each other.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) accomplished this by encouraging alignment of representations of words.
Revised: (2019) accomplished this by encouraging alignment of representations of words.

Original: This was enhanced by Ormazabal et al.
Revised: This was enhanced by Ormazabal et al.

Original: (2021) via a mechanism (Cross-lingual Anchoring) that extended the skip-gram to have bilingual target embeddings.
Revised: (2021) via a mechanism (Cross-lingual Anchoring) that extended the skip-gram to have bilingual target embeddings.

Original: Ultimately, these can learn a crosslingual embedding that preserves synonyms, and is optimized over the entire source language data.
Revised: Ultimately, these can learn a crosslingual embedding that preserves synonyms, and is optimized over the entire source language data.

Original: Results.
Revised: Results.

Original: Both Ormazabal et al.
Revised: Both Ormazabal et al.

Original: (2021) and Wang et al.
Revised: (2021) and Wang et al.

Original: (2019) reported P@1 (precision of the 1st nearest neighbor) BLI results for their method – as opposed to the more robust MAP evaluation we use in the remainder of the paper.
Revised: (2019) reported P@1 (precision of the 1st nearest neighbor) BLI results for their method – as opposed to the more robust MAP evaluation we use in the remainder of the paper.

Original: So we report the BLI performance on the MUSE dataset onlanguages before and after I-C+SN+L for a variety of alignment techniques in Table 16.
Revised: So we report the BLI performance on the MUSE dataset onlanguages before and after I-C+SN+L for a variety of alignment techniques in Table 16.

Original: We include their published numbers directly for comparison.
Revised: We include their published numbers directly for comparison.

Original: We also include reported numbers from various other papers, denoted with ∗ .
Revised: We also include reported numbers from various other papers, denoted with ∗ .

Original: The work of Ormazabal et al.
Revised: The work of Ormazabal et al.

Original: (2021) slightly outperforms I-C+SN+L (with RCSLS (Joulin et al., 2018)) on EN-DE, EN-ES, and EN-FR, but under-performs on EN-RU and as a result has a lower overall average.
Revised: (2021) slightly outperforms I-C+SN+L (with RCSLS (Joulin et al., 2018)) on EN-DE, EN-ES, and EN-FR, but under-performs on EN-RU and as a result has a lower overall average.

Original: The result of Ormazabal et al.
Revised: The result of Ormazabal et al.

Original: (2021) are certainly intriguing, and we hope to be able to compare on a broader set of languages and use the MAP evaluation in future work.
Revised: (2021) are certainly intriguing, and we hope to be able to compare on a broader set of languages and use the MAP evaluation in future work.

Original: We remark, however, that such approaches are less flexible than using a post-processing alignment as they require re-training, so it is less clear how to apply for multiple languages, would need to be re-trained for other languages pairs (e.g., DE-FR), and how it will work for lower resource languages where less training data is available.
Revised: We remark, however, that such approaches are less flexible than using a post-processing alignment as they require re-training, so it is less clear how to apply for multiple languages, would need to be re-trained for other languages pairs (e.g., DE-FR), and how it will work for lower resource languages where less training data is available.

Original: After applying our proposed I-C+SN+L normalization, ADV (Conneau et al., 2018b) failed to align
Revised: After applying our proposed I-C+SN+L normalization, ADV (Conneau et al., 2018b) failed to align

Original: EN-DE, EN-ES, EN-RU.
Revised: EN-DE, EN-ES, EN-RU.

Original: This method uses a Generative Adversarial Network (GAN), and this phenomenon is explained by Hartmann et al.
Revised: This method uses a Generative Adversarial Network (GAN), and this phenomenon is explained by Hartmann et al.

Original: In their works, they showed that unit length normalization makes GAN based CLWE method unstable and deteriorates their performance but other supervised alignments are not affected by it.
Revised: In their works, they showed that unit length normalization makes GAN based CLWE method unstable and deteriorates their performance but other supervised alignments are not affected by it.

Original: This explains why the performance on PROC (Conneau et al., 2018a), RCSLS (Joulin et al., 2018) and Wasserstein Procrustes (WP) (Grave et al., 2019) are not affected by I-C+SN+L, and in fact, it leads to an improvement in the BLI performance.
Revised: This explains why the performance on PROC (Conneau et al., 2018a), RCSLS (Joulin et al., 2018) and Wasserstein Procrustes (WP) (Grave et al., 2019) are not affected by I-C+SN+L, and in fact, it leads to an improvement in the BLI performance.

Original: G A LIGNMENT OF S INGLE - CELL M ULTI - OMICS D ATA
Revised: G A LIGNMENT OF S INGLE - CELL M ULTI - OMICS D ATA

Original: [Table]
Revised: [Table]

Original: Single-cell measurements allow scientists to study the various properties of the genome such as gene expression, chromatin accessibility, DNA methylation etc.
Revised: Single-cell measurements allow scientists to study the various properties of the genome such as gene expression, chromatin accessibility, DNA methylation etc.

Original: Therefore there is the need for data integration of these single-cell measurements so that scientists can understand cell development over time and disease.
Revised: Therefore there is the need for data integration of these single-cell measurements so that scientists can understand cell development over time and disease.

Original: However, due to the lack of cell-to-cell correspondence between the different types of measurements, it makes the process of data integration a challenging task.
Revised: However, due to the lack of cell-to-cell correspondence between the different types of measurements, it makes the process of data integration a challenging task.

Original: Motivated by this, (Demetci et al., 2020) proposed a Single-Cell alignment using Optimal Transport (SCOT), an unsupervised learning algorithm that uses Gromov Wasserstein-based optimal transport to recover the cell-to-cell correspondence between two sequencing domains.
Revised: Motivated by this, (Demetci et al., 2020) proposed a Single-Cell alignment using Optimal Transport (SCOT), an unsupervised learning algorithm that uses Gromov Wasserstein-based optimal transport to recover the cell-to-cell correspondence between two sequencing domains.

Original: * represents results taken directly from (Demetci et al., 2020).
Revised: * represents results taken directly from (Demetci et al., 2020).

Original: MMD-MA (Liu et al., 2019) and UnionCom (Cao et al., 2020) are two other unsupervised single-cell alignment methods.
Revised: MMD-MA (Liu et al., 2019) and UnionCom (Cao et al., 2020) are two other unsupervised single-cell alignment methods.

Original: We follow the implementation details from (Demetci et al., 2020).
Revised: We follow the implementation details from (Demetci et al., 2020).

Original: * represents results taken directly from (Demetci et al., 2020).
Revised: * represents results taken directly from (Demetci et al., 2020).

Original: We use two single-cell multi-omics datasets: 1) scGEM assay (Cheow et al., 2016) and 2) SNAREseq assay (Chen et al., 2019).
Revised: We use two single-cell multi-omics datasets: 1) scGEM assay (Cheow et al., 2016) and 2) SNAREseq assay (Chen et al., 2019).

Original: The fraction of samples is closer than the true match, (FOSCTTM) is used to evaluate the alignment performance.
Revised: The fraction of samples is closer than the true match, (FOSCTTM) is used to evaluate the alignment performance.

Original: A lower average FOSCTTM score indicates a higher ability of SCOT to recover the correct correspondences.
Revised: A lower average FOSCTTM score indicates a higher ability of SCOT to recover the correct correspondences.

Original: Table 17 summaries the before and after I-C+L and I-C+SN+L SCOT performance based on FOSCTTM.
Revised: Table 17 summaries the before and after I-C+L and I-C+SN+L SCOT performance based on FOSCTTM.

Original: On the SNAREseq dataset SCOT with I-C+SN+L outperforms SCOT with I-C+L but achieves the same performance as SCOT without normalization.
Revised: On the SNAREseq dataset SCOT with I-C+SN+L outperforms SCOT with I-C+L but achieves the same performance as SCOT without normalization.

Original: However, processing the scGEM dataset with I-C+SN+L before alignment achieved the highest FOSCTTM score of 0.184 followed by SCOT without normalization.
Revised: However, processing the scGEM dataset with I-C+SN+L before alignment achieved the highest FOSCTTM score of 0.184 followed by SCOT without normalization.

Original: H C ODE FOR S PEC N ORM
Revised: H C ODE FOR S PEC N ORM

Original: The code implementing our new algorithm Spectral Normalization is provided here https:// github.com/poaboagye/SpecNorm .
Revised: The code implementing our new algorithm Spectral Normalization is provided here https:// github.com/poaboagye/SpecNorm .

Original: As it is quite simple, for completeness we also present it next.
Revised: As it is quite simple, for completeness we also present it next.

Original: Spectral Normalization (SpecNorm) is below, referred to as SpecNorm.py .
Revised: Spectral Normalization (SpecNorm) is below, referred to as SpecNorm.py .

Original: Similarly, the code implementation of Iterative Spectral Normalization (I-C+SN+L) referred to as SpecNorm usage.py is also shown below.
Revised: Similarly, the code implementation of Iterative Spectral Normalization (I-C+SN+L) referred to as IterSpecNorm.py is also shown below.

Original: Spectral Normalization (SpecNorm.py) import numpy as np def computeSVD(embeds): """
Revised: Spectral Normalization (SpecNorm.py) import numpy as np def computeSVD(embed): """

Original: Args:embeds: Monolingual Embedding
Revised: Args:emded: Monolingual Embedding

Original: Returns:Singular Value Decomposition """ U, S, VT = np.linalg.svd(embeds,full_matrices=
Revised: Returns:Singular Value Decomposition """ U, S, VT = np.linalg.svd(embed,full_matrices=

Original: False) return U, S, VT def SVDConX(embeds, beta): """
Revised: False) return U, S, VT def specNorm(embed, beta): """

Original: Args:embeds: Monolingual Embedding beta: Use to determine smaller (noisy) singular values to be removed
Revised: Args:emded: Monolingual Embedding beta: Use to determine smaller (noisy) singular values to be removed

Original: Returns:Spectral Normalised Embedding """#Perform SVD on the Monolingual Embedding _, S, VT = computeSVD(embeds)
Revised: Returns:Spectral Normalized Embedding """#Perform SVD on the Monolingual Embedding _, S, VT = computeSVD(embed)

Original: # Compute eta eta = np.sqrt(np.sum(S**2)/len(S))
Revised: # Compute eta eta = np.sqrt(np.sum(S**2)/len(S))

Original: # Transform diagonal matrix S_prime = 1 / S for idx, sigma in enumerate(S):
Revised: # Transform diagonal matrix S_prime = 1 / S for idx, sigma in enumerate(S):

Original: if sigma > beta*eta:
Revised: if sigma > beta*eta:

Original: S_prime[idx] = S_prime[idx] * (beta*eta) else :
Revised: S_prime[idx] = S_prime[idx] * (beta*eta) else :

Original: S_prime[idx] = 1 S_prime = np.eye(len(S))
Revised: S_prime[idx] = 1 S_prime = np.eye(len(S))

Original: * S_prime # Compute new monolingual embedding embeds = embeds @ VT.T @
Revised: * S_prime # Compute new monolingual embedding embed = embed @ VT.T @

Original: S_prime return embeds
Revised: S_prime return embed

Original: Iterative Spectral Normalization (SpecNorm usage.py) import numpy as np from SVDConX import SVDConX from argparse import ArgumentParser def load_embed(filename, max_vocab=-1): words, embeds =
Revised: Iterative Spectral Normalization (IterSpecNorm.py) import numpy as np from SpecNorm import specNorm from argparse import ArgumentParser def load_embed(filename, max_vocab=-1): words, embeds =

Original: [], [] with open (filename, ’r’) as f: next (f) for line in f: word, vector = line.rstrip().split(’ ’, 1) vector = np.fromstring(vector, sep=’ ’)
Revised: [], [] with open (filename, ’r’) as f: next (f) for line in f: word, vector = line.rstrip().split(’ ’, 1) vector = np.fromstring(vector, sep=’ ’)

Original: words.append(word)embeds.append(vector)
Revised: words.append(word)embeds.append(vector)

Original: if len(embeds) ==
Revised: if len(embeds) ==

Original: max_vocab: break return words, np.array(embeds) def saveEmbed(path, words, word_embeds): with open (path, ’w’) as f: print (word_embeds.shape[0], word_embeds.shape[1], file =f) for word, embed in zip(words, word_embeds): vector_str = ’ ’.join(str(x)
Revised: max_vocab: break return words, np.array(embeds) def saveEmbed(path, words, word_embeds): with open (path, ’w’) as f: print (word_embeds.shape[0], word_embeds.shape[1], file =f) for word, embed in zip(words, word_embeds): vector_str = ’ ’.join(str(x)

Original: for x in embed) print (word, vector_str, file = f) def main(): parser= ArgumentParser() parser.add_argument(’--input_file’)
Revised: for x in embed) print (word, vector_str, file = f) def main(): parser= ArgumentParser() parser.add_argument(’--input_file’)

Original: parser.add_argument(’--output_file’)parser.add_argument(’--niter’,default=5, type= int ) parser.add_argument(’--max_vocab’, default=200000, type= int ) parser.add_argument(’--beta’, default=2, type= int ) args = parser.parse_args() words, embeds = load_embed(args.input_file, max_vocab=args.max_vocab) embeds /= np.linalg.norm(embeds, axis=1)
Revised: parser.add_argument(’--output_file’)parser.add_argument(’--niter’,default=5, type= int ) parser.add_argument(’--max_vocab’, default=200000, type= int ) parser.add_argument(’--beta’, default=2, type= int ) args = parser.parse_args() words, embeds = load_embed(args.input_file, max_vocab=args.max_vocab) embeds /= np.linalg.norm(embeds, axis=1)

Original: [:, np.newaxis] + 1efor i in range (args.niter):
Revised: [:, np.newaxis] + 1efor i in range (args.niter):

Original: [Table]
Revised: [Table]

Original: if __name__ == ’__main__’: main()
Revised: if __name__ == ’__main__’: main()

Original: I R UNTIME
Revised: I R UNTIME

Original: Most of the alignment algorithms run on a CPU except for VecMap, which requires a GPU for faster computation.
Revised: Most of the alignment algorithms run on a CPU except for VecMap, which requires a GPU for faster computation.

Original: It takes about 91 seconds to run Iterative Spectral Normalization on a CPU with a β = 2 and five iterations.
Revised: It takes about 91 seconds to run Iterative Spectral Normalization on a CPU with a β = 2 and five iterations.

Original: Hardware specifications are NVIDIA GeForce GTX Titan Xp 12GB, AMD Ryzen 7 1700 eight-core processor, and 62.8GB RAM.
Revised: Hardware specifications are NVIDIA GeForce GTX Titan Xp 12GB, AMD Ryzen 7 1700 eight-core processor, and 62.8GB RAM.

Original: All alignment approaches were completed in under 15 minutes, and most less than 5 minutes.
Revised: All alignment approaches were completed in under 15 minutes, and most less than 5 minutes.

Original: Each evaluation (BLI, CLDC, or XNLI) takes under 2 minutes, but the training step for CLDC and XNLI takes about a day each; hence our approach aims only to need to do this once (on a high resource language like English), and then use the faster alignment step to transfer this to other languages.
Revised: Each evaluation (BLI, CLDC, or XNLI) takes under 2 minutes, but the training step for CLDC and XNLI takes about a day each; hence our approach aims only to need to do this once (on a high resource language like English), and then use the faster alignment step to transfer this to other languages.

Original: [Table]
Revised: [Table]

Original: PR 0.
Revised: PR 0.

Original: GeoMediaN 0.
Revised: GeoMediaN 0.

Original: C+L 0.
Revised: C+L 0.

Original: I-C+L 0.
Revised: I-C+L 0.

Original: SN 0.
Revised: SN 0.

Original: C+SN+L 0.
Revised: C+SN+L 0.

Original: I-C+SN+L 0.
Revised: I-C+SN+L 0.

Original: RCSLS
Revised: RCSLS

Original: None 0.
Revised: None 0.

Original: PR 0.
Revised: PR 0.

Original: GeoMediaN 0.
Revised: GeoMediaN 0.

Original: [Table]
Revised: [Table]

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: SN 0.
Revised: SN 0.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: None
Revised: None

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: PR
Revised: PR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: GeoMediaN 0.
Revised: GeoMediaN 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: SN 0.
Revised: SN 0.

Original: [Table]
Revised: [Table]

Original: SN
Revised: SN

Original: [Table] [Table]
Revised: [Table] [Table]

Original: K
Revised: K

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: DLV 5k 0.
Revised: DLV 5k 0.

Original: DLV
Revised: DLV

Original: [Table]
Revised: [Table]

Original: 5k 0.
Revised: 5k 0.

Original: DLV SN 5k 0.
Revised: DLV SN 5k 0.

Original: RCSLS 5k 0.
Revised: RCSLS 5k 0.

Original: RCSLS I-C+L 5k 0.
Revised: RCSLS I-C+L 5k 0.

Original: RCSLS SN 5k 0.
Revised: RCSLS SN 5k 0.

Original: VECMAP0.
Revised: VECMAP0.

Original: VECMAP I-C+L0.
Revised: VECMAP I-C+L0.

Original: VECMAP SN0.
Revised: VECMAP SN0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: RCSLS 5k 0.
Revised: RCSLS 5k 0.

Original: RCSLS I-C+L 5k 0.
Revised: RCSLS I-C+L 5k 0.

Original: RCSLS SN 5k 0.
Revised: RCSLS SN 5k 0.

Original: VECMAP0.
Revised: VECMAP0.

Original: VECMAP I-C+L0.
Revised: VECMAP I-C+L0.

Original: VECMAP SN0.
Revised: VECMAP SN0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: DLV SN 3k 0.
Revised: DLV SN 3k 0.

Original: DLV 5k 0.
Revised: DLV 5k 0.

Original: DLV SN 5k 0.
Revised: DLV SN 5k 0.

Original: RCSLS 1k 0.
Revised: RCSLS 1k 0.

Original: RCSLS SN 1k 0.
Revised: RCSLS SN 1k 0.

Original: RCSLS 3k 0.
Revised: RCSLS 3k 0.

Original: RCSLS SN 3k 0.
Revised: RCSLS SN 3k 0.

Original: RCSLS 5k 0.
Revised: RCSLS 5k 0.

Original: RCSLS SN 5k 0.
Revised: RCSLS SN 5k 0.

Original: VECMAP0.
Revised: VECMAP0.

Original: VECMAP SN0.
Revised: VECMAP SN0.

Original: [Table]
Revised: [Table]

Original: PROC-B SN 1k 0.
Revised: PROC-B SN 1k 0.

Original: PROC-B 3k 0.
Revised: PROC-B 3k 0.

Original: PROC-B SN 3k 0.
Revised: PROC-B SN 3k 0.

Original: DLV 1k 0.
Revised: DLV 1k 0.

Original: DLV SN 1k 0.
Revised: DLV SN 1k 0.

Original: DLV 3k 0.
Revised: DLV 3k 0.

Original: DLV SN 3k 0.
Revised: DLV SN 3k 0.

Original: DLV 5k 0.
Revised: DLV 5k 0.

Original: DLV SN 5k 0.
Revised: DLV SN 5k 0.

Original: RCSLS 1k 0.
Revised: RCSLS 1k 0.

Original: RCSLS SN 1k 0.
Revised: RCSLS SN 1k 0.

Original: RCSLS 3k 0.
Revised: RCSLS 3k 0.

Original: RCSLS SN 3k 0.
Revised: RCSLS SN 3k 0.

Original: RCSLS 5k 0.
Revised: RCSLS 5k 0.

Original: [Table]
Revised: [Table]

Original: VECMAP0.
Revised: VECMAP0.

Original: [Table]
Revised: [Table]