Original: BLISS IN N ON -I SOMETRIC E MBEDDING S PACES
Revised: BLISS IN N ON -I SOMETRIC E MBEDDING S PACES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent work on bilingual lexicon induction (BLI) has frequently depended either on aligned bilingual lexicons or on distribution matching, often with an assumption about the isometry of the two spaces.
Revised: Recent work on bilingual lexicon induction (BLI) has frequently depended either on aligned bilingual lexicons or on distribution matching, often with an assumption about the isometry of the two spaces.

Original: We propose a technique to quantitatively estimate this assumption of the isometry between two embedding spaces and empirically show that this assumption weakens as the languages in question become increasingly etymologically distant.
Revised: We propose a technique to quantitatively estimate this assumption of the isometry between two embedding spaces and empirically show that this assumption weakens as the languages in question become increasingly etymologically distant.

Original: We then propose Bilingual Lexicon Induction with Semi-Supervision (BLISS) — a novel semi-supervised approach that relaxes the isometric assumption while leveraging both limited aligned bilingual lexicons and a larger set of unaligned word embeddings, as well as a novel hubness ﬁltering technique.
Revised: We then propose Bilingual Lexicon Induction with Semi-Supervision (BLISS) — a novel semi-supervised approach that relaxes the isometric assumption while leveraging both limited aligned bilingual lexicons and a larger set of unaligned word embeddings, as well as a novel hubness ﬁltering technique.

Original: Our proposed method improves over strong baselines for 11 of 14 language pairs on the MUSE dataset, particularly for languages whose embedding spaces do not appear to be isometric.
Revised: Our proposed method improves over strong baselines for 11 of 14 language pairs on the MUSE dataset, particularly for languages whose embedding spaces do not appear to be isometric.

Original: In addition, we also show that adding supervision stabilizes the learning procedure, and is effective even with minimal supervision.
Revised: In addition, we also show that adding supervision stabilizes the learning procedure, and is effective even with minimal supervision.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bilingual lexicon induction (BLI), the task of ﬁnding corresponding words in two languages from comparable corpora (Haghighi et al., 2008; Xing et al., 2015; Zhang et al., 2017b; Artetxe et al., 2017; Lample et al., 2018), ﬁnds use in numerous NLP tasks like POS tagging (Zhang et al., 2016), parsing (Xiao & Guo, 2014), document classiﬁcation (Klementiev et al., 2012), and machine translation (Irvine & Callison-Burch, 2013; Qi et al., 2018).
Revised: Bilingual lexicon induction (BLI), the task of ﬁnding corresponding words in two languages from comparable corpora (Haghighi et al., 2008; Xing et al., 2015; Zhang et al., 2017b; Artetxe et al., 2017; Lample et al., 2018), ﬁnds use in numerous NLP tasks like POS tagging (Zhang et al., 2016), parsing (Xiao & Guo, 2014), document classiﬁcation (Klementiev et al., 2012), and machine translation (Irvine & Callison-Burch, 2013; Qi et al., 2018).

Original: Most work on BLI uses methods that learn a mapping between two word embedding spaces (Ruder, 2017), which makes it possible to leverage pre-trained embeddings learned on large monolingual corpora.
Revised: Most work on BLI uses methods that learn a mapping between two word embedding spaces (Ruder, 2017), which makes it possible to leverage pre-trained embeddings learned on large monolingual corpora.

Original: A commonly used method for BLI, which also empirically works well, involves learning an orthogonal mapping between the two embedding spaces (Mikolov et al.
Revised: A commonly used method for BLI, which also empirically works well, involves learning an orthogonal mapping between the two embedding spaces (Mikolov et al.

Original: (2013a), Xing et al.
Revised: (2013a), Xing et al.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2016), Smith et al.
Revised: (2016), Smith et al.

Original: However, learning an orthogonal mapping inherently assumes that the embedding spaces for the two languages are isometric (subsequently referred to as the orthogonality assumption).
Revised: However, learning an orthogonal mapping inherently assumes that the embedding spaces for the two languages are isometric (subsequently referred to as the orthogonality assumption).

Original: This is a particularly strong assumption that may not necessarily hold true, and consequently we can expect methods relying on this assumption to provide sub-optimal results.
Revised: This is a particularly strong assumption that may not necessarily hold true, and consequently we can expect methods relying on this assumption to provide sub-optimal results.

Original: In this work, we examine this assumption, identify where it breaks down, and propose a method to alleviate this problem.
Revised: In this work, we examine this assumption, identify where it breaks down, and propose a method to alleviate this problem.

Original: We ﬁrst formally describe this orthogonality assumption, and then present a theoretically motivated approach based on the Gromov-Hausdroff (GH) distance to check the validity and extent of the orthogonality assumption ( § 2).
Revised: We ﬁrst formally describe this orthogonality assumption, and then present a theoretically motivated approach based on the Gromov-Hausdroff (GH) distance to check the validity and extent of the orthogonality assumption ( § 2).

Original: We show that the constraint indeed does not hold, particularly for etymologically distant language pairs.
Revised: We show that the constraint indeed does not hold, particularly for etymologically distant language pairs.

Original: SpeMotivated by the above observation, we propose a framework for B ilingual L exicon I nduction with S
Revised: SpeMotivated by the above observation, we propose a framework for B ilingual L exicon I nduction with S

Original: emi- S upervision ( BLISS ) ( § 3.2) that alleviates the aforementioned issues.
Revised: emi- S upervision ( BLISS ) ( § 3.2) that alleviates the aforementioned issues.

Original: Moreover, besides addressing the limitations of the orthogonality assumption, the semi-supervised framework also addresses the shortcomings of purely supervised and purely unsupervised methods for BLI ( § 3.1).
Revised: Moreover, besides addressing the limitations of the orthogonality assumption, the semi-supervised framework also addresses the shortcomings of purely supervised and purely unsupervised methods for BLI ( § 3.1).

Original: Our framework jointly optimizes for supervised embedding alignment, unsupervised distribution matching, and a weak orthogonality constraint in the form of a back-translation loss.
Revised: Our framework jointly optimizes for supervised embedding alignment, unsupervised distribution matching, and a weak orthogonality constraint in the form of a back-translation loss.

Original: Our results show that the different losses work in tandem to learn a better mapping than any one could on its own ( § 4.3).
Revised: Our results show that the different losses work in tandem to learn a better mapping than any one could on its own ( § 4.3).

Original: We also show that the proposed framework improves performance over strong baselines on two datasets, particularly for the case of embedding spaces where the orthogonal assumption is not valid.
Revised: We also show that the proposed framework improves performance over strong baselines on two datasets, particularly for the case of embedding spaces where the orthogonal assumption is not valid.

Original: Our analysis ( § 4.3) demonstrates that adding supervision to the learning objective, even in the form of a small seed dictionary, signiﬁcantly improves the stability of the learning procedure.
Revised: Our analysis ( § 4.3) demonstrates that adding supervision to the learning objective, even in the form of a small seed dictionary, signiﬁcantly improves the stability of the learning procedure.

Original: In particular, for cases where either the embedding spaces are far apart according to GH distance or the quality of the original embeddings is poor, our framework converges where the unsupervised baselines fail to.
Revised: In particular, for cases where either the embedding spaces are far apart according to GH distance or the quality of the original embeddings is poor, our framework converges where the unsupervised baselines fail to.

Original: We also show that for the same amount of available supervised data, leveraging unsupervised learning allows us to obtain superior performance over baseline supervised and unsupervised methods using a comparable amount of data.
Revised: We also show that for the same amount of available supervised data, leveraging unsupervised learning allows us to obtain superior performance over baseline supervised and unsupervised methods using a comparable amount of data.

Original: All our code can be found at www.toaddhereifaccepted.com .
Revised: All our code can be found at www.toaddhereifaccepted.com .

Original:  I SOMETRY OF E MBEDDING S PACES
Revised:  I SOMETRY OF E MBEDDING S PACES

Original: Both supervised and unsupervised BLI often rely on the assumption that the word embedding spaces are isometric to each other.
Revised: Both supervised and unsupervised BLI often rely on the assumption that the word embedding spaces are isometric to each other.

Original: Thus, they learn an orthogonal mapping matrix to map one space to another.
Revised: Thus, they learn an orthogonal mapping matrix to map one space to another.

Original: For example, for the case of supervised Bilingual Lexicon Induction, Xing et al.
Revised: For example, for the case of supervised Bilingual Lexicon Induction, Xing et al.

Original: (2015) learn an orthogonal mapping matrix to minimize the distance between the projected source and the target embeddings; while in the unsupervised case, Lample et al.
Revised: (2015) learn an orthogonal mapping matrix to minimize the distance between the projected source and the target embeddings; while in the unsupervised case, Lample et al.

Original: (2018) propose learning a matrix near the manifold of orthogonal matrices to match the distributions of the projected source and target word embeddings.
Revised: (2018) propose learning a matrix near the manifold of orthogonal matrices to match the distributions of the projected source and target word embeddings.

Original: We hypothesize that this assumption might not always hold, in particular for the cases when the language pairs in consideration are etymologically distant — Zhang et al.
Revised: We hypothesize that this assumption might not always hold, in particular for the cases when the language pairs in consideration are etymologically distant — Zhang et al.

Original: (2017a)
Revised: (2017a)

Original: and Søgaard et al.
Revised: and Søgaard et al.

Original: (2018) provide evidence of this by observing a higher Earth Mover’s distance and eigenvector similarity metric respectively between etymologically distant languages.
Revised: (2018) provide evidence of this by observing a higher Earth Mover’s distance and eigenvector similarity metric respectively between etymologically distant languages.

Original: In order to test this hypothesis, we propose a novel way of a-priori analyzing the validity of the orthogonality assumption using the
Revised: In order to test this hypothesis, we propose a novel way of a-priori analyzing the validity of the orthogonality assumption using the

Original: Gromov Hausdorff (also referred as GH) distance.
Revised: Gromov Hausdorff (also referred as GH) distance.

Original: In order to analyze the validity of the orthogonality assumption, we quantitatively check how well the metric spaces of the word vectors of two languages can be aligned under an isometric transform using the GH distance.
Revised: In order to analyze the validity of the orthogonality assumption, we quantitatively check how well the metric spaces of the word vectors of two languages can be aligned under an isometric transform using the GH distance.

Original: The Hausdorff distance between two metric spaces is a measure of the worst case or the diametric distance between the spaces.
Revised: The Hausdorff distance between two metric spaces is a measure of the worst case or the diametric distance between the spaces.

Original: Intuitively, it measures the distance between the nearest neighbours that are the farthest apart.
Revised: Intuitively, it measures the distance between the nearest neighbours that are the farthest apart.

Original: Concretely, given two metric spaces X , and Y with a distance function d ( ., . ) , the Hausdorff distance is deﬁned as:
Revised: Concretely, given two metric spaces X , and Y with a distance function d ( ., . ) , the Hausdorff distance is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The Gromov-Hausdorff distance minimizes the Hausdorff distance over all isometric transforms between X and Y , thereby providing a quantitative estimate of the isometry of two spaces
Revised: The Gromov-Hausdorff distance minimizes the Hausdorff distance over all isometric transforms between X and Y , thereby providing a quantitative estimate of the isometry of two spaces

Original: [Equation]
Revised: [Equation]

Original: where f, g belong to set of isometric transforms.
Revised: where f, g belong to set of isometric transforms.

Original: Computing the Gromov-Hausdorff distance involves solving hard combinatorial problems, but can be tractably approximated using the Bottleneck distance (Chazal et al., 2009).
Revised: Computing the Gromov-Hausdorff distance involves solving hard combinatorial problems, but can be tractably approximated using the Bottleneck distance (Chazal et al., 2009).

Original: In order to compute the Bottleneck distance between two metric spaces, we compute the ﬁrst order Vietoris-Rips complex (ﬁrst order for computational efﬁciency) at t for both spaces: a graph containing an edge between two points iff they lie within a Euclidean distance t from each other in the metric space.
Revised: In order to compute the Bottleneck distance between two metric spaces, we compute the ﬁrst order Vietoris-Rips complex (ﬁrst order for computational efﬁciency) at t for both spaces: a graph containing an edge between two points iff they lie within a Euclidean distance t from each other in the metric space.

Original: As t is varied, the Vietoris-Rips complex goes from the individual points (at t = 0 ) to a single cluster (at t = ∞ ).
Revised: As t is varied, the Vietoris-Rips complex goes from the individual points (at t = 0 ) to a single cluster (at t = ∞ ).

Original: As t increases, clusters are formed (birth) and eventually merge together (death).
Revised: As t increases, clusters are formed (birth) and eventually merge together (death).

Original: The persistence diagram is a 2D plot of the ( t birth , t death ) of each cluster, where t birth and t death are the values of t at which the cluster was born and died respectively.
Revised: The persistence diagram is a 2D plot of the ( t birth , t death ) of each cluster, where t birth and t death are the values of t at which the cluster was born and died respectively.

Original: Given two persistence diagrams f, g , let γ be a bijective map from the points of f to the points of g .
Revised: Given two persistence diagrams f, g , let γ be a bijective map from the points of f to the points of g .

Original: The bottleneck distance ( B ) is then deﬁned as:
Revised: The bottleneck distance ( B ) is then deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Chazal et al.
Revised: Chazal et al.

Original: (2009) showed that the Gromov-Hausdorff distance can be lower bounded by the Bottleneck Distance between the Persistence Diagrams of the Vietoris-Rips Filtration of the two spaces.
Revised: (2009) showed that the Gromov-Hausdorff distance can be lower bounded by the Bottleneck Distance between the Persistence Diagrams of the Vietoris-Rips Filtration of the two spaces.

Original: We compute this lower bound for the top frequency words of different language pairs.
Revised: We compute this lower bound for the top frequency words of different language pairs.

Original: As can be observed from Figure 1, the GH distances are higher for distant language pairs.
Revised: As can be observed from Figure 1, the GH distances are higher for distant language pairs.

Original:  S EMI - SUPERVISED F RAMEWORK
Revised:  S EMI - SUPERVISED F RAMEWORK

Original: In this section, we motivate and deﬁne our semi-supervised framework for BLI.
Revised: In this section, we motivate and deﬁne our semi-supervised framework for BLI.

Original: First we describe issues with purely supervised and unsupervised methods, and then lay the framework for tackling them along with orthogonality constraints.
Revised: First we describe issues with purely supervised and unsupervised methods, and then lay the framework for tackling them along with orthogonality constraints.

Original: 3. D RAWBACKS OF P URELY S UPERVISED AND U NSUPERVISED M ETHODS
Revised: 3. D RAWBACKS OF P URELY S UPERVISED AND U NSUPERVISED M ETHODS

Original: Purely supervised methods for aligning word vectors do not utilize the rich information present in the topology of the word vectors.
Revised: Purely supervised methods for aligning word vectors do not utilize the rich information present in the topology of the word vectors.

Original: Purely unsupervised methods, on the other hand, can suffer from poor performance if the distribution of the embedding spaces of the two languages are very different from each other.
Revised: Purely unsupervised methods, on the other hand, can suffer from poor performance if the distribution of the embedding spaces of the two languages are very different from each other.

Original: Moreover, unsupervised methods can successfully align clusters of words together, but miss out on ﬁne grained alignment within the clusters.
Revised: Moreover, unsupervised methods can successfully align clusters of words together, but miss out on ﬁne grained alignment within the clusters.

Original: We explicitly show the aforementioned problem of purely unsupervised methods with the help of the toy dataset shown in 2a, and 2b.
Revised: We explicitly show the aforementioned problem of purely unsupervised methods with the help of the toy dataset shown in 2a, and 2b.

Original: In this dataset, due to the density difference between the two large blue clusters, unsupervised matching is consistently able to align them properly, but has trouble aligning the smaller embedded green and red sub-clusters.
Revised: In this dataset, due to the density difference between the two large blue clusters, unsupervised matching is consistently able to align them properly, but has trouble aligning the smaller embedded green and red sub-clusters.

Original: The correct transformation of the source space is a clockwise 90 ◦ rotation followed by reﬂection along the x-axis.
Revised: The correct transformation of the source space is a clockwise 90 ◦ rotation followed by reﬂection along the x-axis.

Original: Unsupervised matching converges to this correct transformation only half of the time; in rest of the cases, it ignores the alignment of the sub-clusters and converges to a 90 ◦ counter-clockwise transformation as shown in 2c.
Revised: Unsupervised matching converges to this correct transformation only half of the time; in rest of the cases, it ignores the alignment of the sub-clusters and converges to a 90 ◦ counter-clockwise transformation as shown in 2c.

Original: We also ﬁnd evidence of this problem in the real datasets used in our experiments as shown in
Revised: We also ﬁnd evidence of this problem in the real datasets used in our experiments as shown in

Original: Table 1.
Revised: Table 1.

Original: It can be seen that the unsupervised method aligns clusters of similar words, but is poor at the ﬁne grained alignment.
Revised: It can be seen that the unsupervised method aligns clusters of similar words, but is poor at the ﬁne grained alignment.

Original: We hypothesize that this problem can be resolved by giving it a some supervision in the form of matching anchor points inside these sub-clusters, which correctly aligns them.
Revised: We hypothesize that this problem can be resolved by giving it a some supervision in the form of matching anchor points inside these sub-clusters, which correctly aligns them.

Original: Analogously, for the task of BLI, generating a small supervised seed lexicon is generally feasible for most language pairs, through either bilingual speakers, existing dictionary resources, or
Revised: Analogously, for the task of BLI, generating a small supervised seed lexicon is generally feasible for most language pairs, through either bilingual speakers, existing dictionary resources, or

Original: Wikipedia language links.
Revised: Wikipedia language links.

Original: This can provide the requisite supervision.
Revised: This can provide the requisite supervision.

Original: 3. A S EMI - SUPERVISED F RAMEWORK
Revised: 3. A S EMI - SUPERVISED F RAMEWORK

Original: In order to alleviate the problems with the orthogonality constraints, the purely unsupervised and supervised approaches, we propose a semi-supervised framework, described below.
Revised: In order to alleviate the problems with the orthogonality constraints, the purely unsupervised and supervised approaches, we propose a semi-supervised framework, described below.

Original: Let X = { x 1 . . .
Revised: Let X = { x 1 . . .

Original: x n } and Y = { y 1 . . .
Revised: x n } and Y = { y 1 . . .

Original: y m } , x i , y i ∈
Revised: y m } , x i , y i ∈

Original: R
Revised: R

Original: d be two sets of word embeddings from the source and target language respectively and let S = { ( x s 1 , y s 1 ) . . .
Revised: d be two sets of word embeddings from the source and target language respectively and let S = { ( x s 1 , y s 1 ) . . .

Original: ( x sk , y sk ) } denote the bilingual aligned word embeddings.
Revised: ( x sk , y sk ) } denote the bilingual aligned word embeddings.

Original: Deﬁne W to be the mapping matrix.
Revised: Deﬁne W to be the mapping matrix.

Original: [Figure]
Revised: [Figure]

Original: For learning W , we leverage unsupervised distribution matching, aligning known word pairs and a data-driven weak orthogonality constraint.
Revised: For learning W , we leverage unsupervised distribution matching, aligning known word pairs and a data-driven weak orthogonality constraint.

Original: Unsupervised Distribution Matching :
Revised: Unsupervised Distribution Matching :

Original: Given all word embeddings X and Y , the unsupervised loss
Revised: Given all word embeddings X and Y , the unsupervised loss

Original: L W | D aims to match the distribution of both embedding spaces.
Revised: L W | D aims to match the distribution of both embedding spaces.

Original: In particular, for our formulation, we use an adversarial distribution matching objective, similar to the work of Lample et al.
Revised: In particular, for our formulation, we use an adversarial distribution matching objective, similar to the work of Lample et al.

Original: Speciﬁcally, a mapping matrix W from the source to the target is learned to fool a discriminator D , which is trained to distinguish between the mapped source embeddings W X
Revised: Speciﬁcally, a mapping matrix W from the source to the target is learned to fool a discriminator D , which is trained to distinguish between the mapped source embeddings W X

Original: = { Wx 1 . . .
Revised: = { Wx 1 . . .

Original: Wx n } and Y .
Revised: Wx n } and Y .

Original: The corresponding objectives are deﬁned as:
Revised: The corresponding objectives are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Aligning Known Word Pairs :
Revised: Aligning Known Word Pairs :

Original: Given aligned bilingual word embeddings S , we aim to minimize a similarity function ( f s ) which maximizes the similarity between the corresponding matched pairs of words.
Revised: Given aligned bilingual word embeddings S , we aim to minimize a similarity function ( f s ) which maximizes the similarity between the corresponding matched pairs of words.

Original: Speciﬁcally, the loss is deﬁned as:
Revised: Speciﬁcally, the loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Weak Orthogonality Constraint :
Revised: Weak Orthogonality Constraint :

Original: Given an embedding space X , we deﬁne a consistency loss that maximizes the similarity f a between x and W T Wx , x ∈ X .
Revised: Given an embedding space X , we deﬁne a consistency loss that maximizes the similarity f a between x and W T Wx , x ∈ X .

Original: This cyclic consistency loss L W|O encourages orthogonality of the W matrix based on the joint optimization:
Revised: This cyclic consistency loss L W|O encourages orthogonality of the W matrix based on the joint optimization:

Original: [Equation]
Revised: [Equation]

Original: The above loss term allows the model adjust the trade-off between orthogonality for accuracy, based on the joint optimization.
Revised: The above loss term allows the model adjust the trade-off between orthogonality for accuracy, based on the joint optimization.

Original: This is particularly helpful for cases where the orthogonality constraint is violated, where the embedding spaces are not isometric, speciﬁcally for etymologically distant language pairs, as we show in (4.3).
Revised: This is particularly helpful for cases where the orthogonality constraint is violated, where the embedding spaces are not isometric, speciﬁcally for etymologically distant language pairs, as we show in (4.3).

Original: The ﬁnal loss function for the mapping matrix is:
Revised: The ﬁnal loss function for the mapping matrix is:

Original: [Equation]
Revised: [Equation]

Original: L W | D enables the model to leverage the distributional information available from the two embedding spaces, thereby using all available monolingual data.
Revised: L W | D enables the model to leverage the distributional information available from the two embedding spaces, thereby using all available monolingual data.

Original: On the other hand, L W | S allows for the correct alignment of labeled pairs when available in the form of a small seed dictionary.
Revised: On the other hand, L W | S allows for the correct alignment of labeled pairs when available in the form of a small seed dictionary.

Original: Finally, L W | O encourages orthogonality.
Revised: Finally, L W | O encourages orthogonality.

Original: One can think of L W | O and L W | S as working against each other when the spaces are not isometric.
Revised: One can think of L W | O and L W | S as working against each other when the spaces are not isometric.

Original: Jointly optimizing both helps the model to strike a balance between them in a data driven manner, encouraging orthogonality but still allowing for ﬂexible mapping.
Revised: Jointly optimizing both helps the model to strike a balance between them in a data driven manner, encouraging orthogonality but still allowing for ﬂexible mapping.

Original: After learning the ﬁnal mapping matrix, the translations of the words in the source language are mapped to the target space and their nearest neighbors according to the CSLS distance (Lample et al., 2018) metric are chosen as the translations.
Revised: After learning the ﬁnal mapping matrix, the translations of the words in the source language are mapped to the target space and their nearest neighbors according to the CSLS distance (Lample et al., 2018) metric are chosen as the translations.

Original: 3. I TERATIVE P ROCRUSTES R EFINEMENT AND H UBNESS R EMOVAL
Revised: 3. I TERATIVE P ROCRUSTES R EFINEMENT AND H UBNESS R EMOVAL

Original: A common method of improving BLI is iteratively expanding the dictionary and learning the matrix as a post-processing step (Artetxe et al., 2017; Lample et al., 2018).
Revised: A common method of improving BLI is iteratively expanding the dictionary and learning the matrix as a post-processing step (Artetxe et al., 2017; Lample et al., 2018).

Original: Given a learnt mapping matrix,
Revised: Given a learnt mapping matrix,

Original: Procrustes Reﬁnement ﬁrst ﬁnds the pair of points in the two languages that are very closely matched by the mapping matrix and constructs a bilingual dictionary from these pairs.
Revised: Procrustes Reﬁnement ﬁrst ﬁnds the pair of points in the two languages that are very closely matched by the mapping matrix and constructs a bilingual dictionary from these pairs.

Original: These pair of points are found by considering the nearest neighbors (NN) of the projected source words in the target space.
Revised: These pair of points are found by considering the nearest neighbors (NN) of the projected source words in the target space.

Original: The mapping matrix is then reﬁned by setting it to be the Procrustes solution of the dictionary obtained.
Revised: The mapping matrix is then reﬁned by setting it to be the Procrustes solution of the dictionary obtained.

Original: Iterative Procrustes Reﬁnement (also referred as Iterative Dictionary Expansion) applies the above step iteratively until some metric (CSLS in case of Lample et al.
Revised: Iterative Procrustes Reﬁnement (also referred as Iterative Dictionary Expansion) applies the above step iteratively until some metric (CSLS in case of Lample et al.

Original: (2018)) converges.
Revised: (2018)) converges.

Original: However, learning an orthogonal linear map in such a way leads to some words (known as hubs) to become nearest neighbors of a majority of other words (Radovanovi ´ c et al., 2010; Dinu & Baroni, 2014).
Revised: However, learning an orthogonal linear map in such a way leads to some words (known as hubs) to become nearest neighbors of a majority of other words (Radovanovi ´ c et al., 2010; Dinu & Baroni, 2014).

Original: In order to estimate the hubness of a point, Radovanovi ´ c et al.
Revised: In order to estimate the hubness of a point, Radovanovi ´ c et al.

Original: ﬁrst computed the distribution N x ( k ) , the counts of all points y such that x ∈ k − NN ( y ) , normalized over all k. The skewness of this distribution was deﬁned as the hubness of the point, with positive skew representing hubs and negative skew representing isolated points.
Revised: ﬁrst computed the distribution N x ( k ) , the counts of all points y such that x ∈ k − NN ( y ) , normalized over all k. The skewness of this distribution was deﬁned as the hubness of the point, with positive skew representing hubs and negative skew representing isolated points.

Original: An approximation to this would be N x (1) , i.e the number of points that x is the 1-NN of.
Revised: An approximation to this would be N x (1) , i.e the number of points that x is the 1-NN of.

Original: We use a simple hubness ﬁltering mechanism to ﬁlter out words in the target domain that are hubs, i.e., words in the target domain which have more than a threshold number of neighbors in the source domain are not considered in the iterative dictionary expansion.
Revised: We use a simple hubness ﬁltering mechanism to ﬁlter out words in the target domain that are hubs, i.e., words in the target domain which have more than a threshold number of neighbors in the source domain are not considered in the iterative dictionary expansion.

Original: Empirically, this leads to a small boost in performance.
Revised: Empirically, this leads to a small boost in performance.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: In this section, we measure the GH distances between embedding spaces of various language pairs, and compute their correlation with several empirical measures of orthogonality.
Revised: In this section, we measure the GH distances between embedding spaces of various language pairs, and compute their correlation with several empirical measures of orthogonality.

Original: Next, we analyze the performance of the instantiations of our semi-supervised framework for two settings of supervised losses, and show that they outperform their supervised and unsupervised counterparts for a majority of the language pairs.
Revised: Next, we analyze the performance of the instantiations of our semi-supervised framework for two settings of supervised losses, and show that they outperform their supervised and unsupervised counterparts for a majority of the language pairs.

Original: Finally we analyze our performance with varying amounts of supervision and highlight the framework’s training stability over unsupervised methods.
Revised: Finally we analyze our performance with varying amounts of supervision and highlight the framework’s training stability over unsupervised methods.

Original: 4. GH D ISTANCE
Revised: 4. GH D ISTANCE

Original: To evaluate the lower bound on the GH distance between the two embedding spaces, we select the 5000 most frequent words of the source and target language and compute the Vietoris-Rips complex of the zeroth order (which considers only pairwise edges for identifying clusters).
Revised: To evaluate the lower bound on the GH distance between the two embedding spaces, we select the 5000 most frequent words of the source and target language and compute the Vietoris-Rips complex of the zeroth order (which considers only pairwise edges for identifying clusters).

Original: These embeddings are mean centered, unit normed and the Euclidean Distance is used as the distance metric.
Revised: These embeddings are mean centered, unit normed and the Euclidean Distance is used as the distance metric.

Original: Row 1 of Table 2 summarizes the GH distances obtained for different language pairs.
Revised: Row 1 of Table 2 summarizes the GH distances obtained for different language pairs.

Original: We ﬁnd that etymologically close languages such as en-fr and ru-uk have a very low GH distance and can possibly be aligned well using orthogonal transforms.
Revised: We ﬁnd that etymologically close languages such as en-fr and ru-uk have a very low GH distance and can possibly be aligned well using orthogonal transforms.

Original: In contrast, we ﬁnd that etymologically distant language pairs such as en-ru and en-hi cannot be aligned well using orthogonal transforms.
Revised: In contrast, we ﬁnd that etymologically distant language pairs such as en-ru and en-hi cannot be aligned well using orthogonal transforms.

Original: [Table]
Revised: [Table]

Original: To further corroborate this, similar to Søgaard et al.
Revised: To further corroborate this, similar to Søgaard et al.

Original: , we compute correlations of the GH distance with the accuracies of several methods for BLI.
Revised: , we compute correlations of the GH distance with the accuracies of several methods for BLI.

Original: We ﬁnd that the GH distance exhibits a strong negative correlation with these accuracies, implying that as the GH distance increases, it becomes increasingly difﬁcult to align these language pairs.
Revised: We ﬁnd that the GH distance exhibits a strong negative correlation with these accuracies, implying that as the GH distance increases, it becomes increasingly difﬁcult to align these language pairs.

Original: Søgaard et al.
Revised: Søgaard et al.

Original: (2018) also proposed the eigenvector similarity metric between embedding spaces for measuring similarity between the embedding spaces.
Revised: (2018) also proposed the eigenvector similarity metric between embedding spaces for measuring similarity between the embedding spaces.

Original: We compute their metric over top n (100, 500, 1000, 5000 and 10000) embeddings (Column Λ in Table 2 shows correlation for the best setting of n ) and show that the GH distance (Column GH) correlates better with the accuracies than eigenvector similarity.
Revised: We compute their metric over top n (100, 500, 1000, 5000 and 10000) embeddings (Column Λ in Table 2 shows correlation for the best setting of n ) and show that the GH distance (Column GH) correlates better with the accuracies than eigenvector similarity.

Original: Furthermore, we also compute correlations against an empirical measure of the orthogonality of two embedding spaces by computing ||
Revised: Furthermore, we also compute correlations against an empirical measure of the orthogonality of two embedding spaces by computing ||

Original: I − W T W || 2 , where W is a mapping from one language to the other obtained from an unsupervised method (MUSE(U)).
Revised: I − W T W || 2 , where W is a mapping from one language to the other obtained from an unsupervised method (MUSE(U)).

Original: Note that an advantage of this metric is that it can be computed even when the supervised dictionaries are not available (ru-uk in Table 2).
Revised: Note that an advantage of this metric is that it can be computed even when the supervised dictionaries are not available (ru-uk in Table 2).

Original: We obtain a strong correlation with this metric as well.
Revised: We obtain a strong correlation with this metric as well.

Original: 4. P ERFORMANCE OF BLISS ON B ENCHMARK TASKS 4.2. B ASELINE M ETHODS
Revised: 4. P ERFORMANCE OF BLISS ON B ENCHMARK TASKS 4.2. B ASELINE M ETHODS

Original: MUSE (U/S/R) (Lample et al., 2018) proposed two models: MUSE(U) and MUSE(S) for unsupervised and supervised BLI respectively.
Revised: MUSE (U/S/R) (Lample et al., 2018) proposed two models: MUSE(U) and MUSE(S) for unsupervised and supervised BLI respectively.

Original: MUSE(U) uses a GAN based distribution matching followed by iterative Procrustes reﬁnement.
Revised: MUSE(U) uses a GAN based distribution matching followed by iterative Procrustes reﬁnement.

Original: MUSE(S) learns an orthogonal map between the embedding spaces by minimizing the Euclidean distance between the supervised translation pairs.
Revised: MUSE(S) learns an orthogonal map between the embedding spaces by minimizing the Euclidean distance between the supervised translation pairs.

Original: Note that for unit normed embedding spaces, this is equivalent to maximizing the cosine similarity between these pairs.
Revised: Note that for unit normed embedding spaces, this is equivalent to maximizing the cosine similarity between these pairs.

Original: MUSE(R) is the semi-supervised extension of MUSE(S), which uses iterative reﬁnement using the CSLS metric starting from the mapping learnt by MUSE(S).
Revised: MUSE(R) is the semi-supervised extension of MUSE(S), which uses iterative reﬁnement using the CSLS metric starting from the mapping learnt by MUSE(S).

Original: We also use our proposed hubness ﬁltering technique during the iterative reﬁnement process (MUSE(HR)) which leads to small performance improvements.
Revised: We also use our proposed hubness ﬁltering technique during the iterative reﬁnement process (MUSE(HR)) which leads to small performance improvements.

Original: We consequently use the hubness ﬁltering technique in all our models.
Revised: We consequently use the hubness ﬁltering technique in all our models.

Original: RCSLS (Joulin et al., 2018) propose optimizing the CSLS metric 2 directly for the supervised matching pairs.
Revised: RCSLS (Joulin et al., 2018) propose optimizing the CSLS metric 2 directly for the supervised matching pairs.

Original: This leads to signiﬁcant improvements over MUSE(S) and achieves state of the art results for a majority of the language pairs at the time of writing.
Revised: This leads to signiﬁcant improvements over MUSE(S) and achieves state of the art results for a majority of the language pairs at the time of writing.

Revised: Vecmap models (Artetxe et al., 2017) and (Artetxe et al., 2018a) proposed two models, Vecmap and

Revised: Vecmap ++ which were based on iterative Procrustes reﬁnement starting from a small seed lexicon based on numeral matching.

Original: 4.2. BLISS MODELS
Revised: 4.2. BLISS MODELS

Original: We instantiate two instances of our framework corresponding to the two supervised losses in the baseline methods mentioned above.
Revised: We instantiate two instances of our framework corresponding to the two supervised losses in the baseline methods mentioned above.

Original: BLISS(M) optimizes the cosine distance between supervised matching pairs as its supervised loss ( L W | S ), while BLISS(R) optimizes the CSLS metric between these matching pairs for its L W | S .
Revised: BLISS(M) optimizes the cosine distance between supervised matching pairs as its supervised loss ( L W | S ), while BLISS(R) optimizes the CSLS metric between these matching pairs for its L W | S .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: We evaluate our models against baselines on two popularly used datasets: the MUSE dataset and the
Revised: We evaluate our models against baselines on two popularly used datasets: the MUSE dataset and the

Original: Vecmap dataset.
Revised: Vecmap dataset.

Original: The MUSE dataset used by Lample et al.
Revised: The MUSE dataset used by Lample et al.

Original: (2018) consists of embeddings trained by Bojanowski et al.
Revised: (2018) consists of embeddings trained by Bojanowski et al.

Original: (2016) on Wikipedia and bilingual dictionaries generated by internal translation tools used at Facebook.
Revised: (2016) on Wikipedia and bilingual dictionaries generated by internal translation tools used at Facebook.

Original: The Vecmap dataset introduced by Dinu & Baroni (2014) consists of the CBOW embeddings trained on the WacKy crawling corpora.
Revised: The Vecmap dataset introduced by Dinu & Baroni (2014) consists of the CBOW embeddings trained on the WacKy crawling corpora.

Original: The bilingual dictionaries were obtained from the Europarl word alignments.
Revised: The bilingual dictionaries were obtained from the Europarl word alignments.

Original: We use the standard training and test splits available for all the bilingual dictionaries in all our experiments.
Revised: We use the standard training and test splits available for all the bilingual dictionaries in all our experiments.

Original: 4.2. R ESULTS
Revised: 4.2. R ESULTS

Original: Table 3 shows the performance of BLISS (M) and BLISS (R) against its supervised and unsupervised counterparts on the MUSE datasets.
Revised: Table 3 shows the performance of BLISS (M) and BLISS (R) against its supervised and unsupervised counterparts on the MUSE datasets.

Original: As can be seen, the semi-supervised framework outperforms the both methods for 9 of 10 language pairs for either cases.
Revised: As can be seen, the semi-supervised framework outperforms the both methods for 9 of 10 language pairs for either cases.

Original: Similar to MUSE datasets, our semi-supervised frameworks outperforms their supervised and unsupervised counterparts on the Vecmap datasets too.
Revised: Similar to MUSE datasets, our semi-supervised frameworks outperforms their supervised and unsupervised counterparts on the Vecmap datasets too.

Original: Table 4 shows the performance of the different models on the Vecmap datasets.
Revised: Table 4 shows the performance of the different models on the Vecmap datasets.

Original: It can be seen that BLISS(M) and BLISS(R) perform better than the MUSE baselines (MUSE(U), MUSE(R)) and RCSLS respectively.
Revised: It can be seen that BLISS(M) and BLISS(R) perform better than the MUSE baselines (MUSE(U), MUSE(R)) and RCSLS respectively.

Original: Note that we also compare against GeoMM (Jawanpuria et al., 2018) and Vecmap ( U )
Revised: Note that we also compare against GeoMM (Jawanpuria et al., 2018) and Vecmap ( U )

Original: +
Revised: +

Original: + (Artetxe et al., 2018b).
Revised: + (Artetxe et al., 2018b).

Original: These methods learn orthogonal mappings for both source and target to a common embedding space, and subsequently do the translations in the common space.
Revised: These methods learn orthogonal mappings for both source and target to a common embedding space, and subsequently do the translations in the common space.

Original: We believe that their superior performance  can be attributed to the fact that the translations are done in the common space, as also investigated in (Kementchedjhieva et al., 2018).
Revised: We believe that their superior performance (particularly on the VecMap datasets) can be attributed to the fact that the translations are done in the common space, as also investigated in (Kementchedjhieva et al., 2018).

Original: Our method suffers from a slight disadvantage against these baselines since we do the translations in the target space, nevertheless we include these baselines for completeness.
Revised: Our method suffers from a slight disadvantage against these baselines since we do the translations in the target space, nevertheless we include these baselines for completeness.

Original: All the hyperparameters for the experiments can be found in the Appendix ( § 7.4)
Revised: All the hyperparameters for the experiments can be found in the Appendix ( § 7.4)

Original: [Figure]
Revised: [Figure]

Original: 4. B ENEFITS OF BLISS
Revised: 4. B ENEFITS OF BLISS

Original: Languages with high GH distance BLISS particularly shines over its supervised counterpart when the two embedding spaces are signiﬁcantly different and the orthogonality constraint is violated.
Revised: Languages with high GH distance BLISS particularly shines over its supervised counterpart when the two embedding spaces are signiﬁcantly different and the orthogonality constraint is violated.

Original: Table 2 shows that BLISS (R) achieves performance gains over RSCLS for language pairs with high
Revised: Table 2 shows that BLISS (R) achieves performance gains over RSCLS for language pairs with high

Original: GH distance.
Revised: GH distance.

Original: Performance with varying amount of supervision Table 5 shows the performance of BLISS(R) as a function of the number of data points provided for supervision.
Revised: Performance with varying amount of supervision Table 5 shows the performance of BLISS(R) as a function of the number of data points provided for supervision.

Original: As can be observed, the model performs reasonably well even for low amounts of supervision and outperforms the unsupervised baseline
Revised: As can be observed, the model performs reasonably well even for low amounts of supervision and outperforms the unsupervised baseline

Original: MUSE(U)
Revised: MUSE(U)

Original: and it’s supervised counterpart
Revised: and it’s supervised counterpart

Original: RCSLS.
Revised: RCSLS.

Original: Moreover, note that the difference is more prominent for en ↔ zh, whose spaces are not isometric, as can be seen from the GH distance.
Revised: Moreover, note that the difference is more prominent for en ↔ zh, whose spaces are not isometric, as can be seen from the GH distance.

Original: In this case the baseline models completely fail to train for 50 points, whereas BLISS(R) performs reasonably well.
Revised: In this case the baseline models completely fail to train for 50 points, whereas BLISS(R) performs reasonably well.

Revised: For a detailed ablation, please refer to

Revised: Appendix 8.

Original: Stability of Training
Revised: Stability of Training

Original: We also observe that providing even a little bit of supervision helps stabilize the training process, when compared to purely unsupervised distribution matching.
Revised: We also observe that providing even a little bit of supervision helps stabilize the training process, when compared to purely unsupervised distribution matching.

Original: [Table]
Revised: [Table]

Original: We measure the stability during training using both the ground truth accuracy and the unsupervised
Revised: We measure the stability during training using both the ground truth accuracy and the unsupervised

Original: CSLS metric (which Lample et al.
Revised: CSLS metric (which Lample et al.

Original: (2018) showed to be correlated with the ground truth accuracy).
Revised: (2018) showed to be correlated with the ground truth accuracy).

Original: As can be seen from Figure 3, BLISS(M) is signiﬁcantly more stable than MUSE(U), converging to better accuracy and CSLS values.
Revised: As can be seen from Figure 3, BLISS(M) is signiﬁcantly more stable than MUSE(U), converging to better accuracy and CSLS values.

Original: When the word vectors are not rich enough (word2vec (Mikolov et al., 2013b) instead of fastText) [Table] the unsupervised method can completely fail to train.
Revised: When the word vectors are not rich enough (word2vec (Mikolov et al., 2013b) instead of fastText), the unsupervised method can completely fail to train.

Original: This can be observed for the case of en-de in
Revised: This can be observed for the case of en-de in

Original: Table 4. BLISS(M) does not face this problem: adding supervision, even in the form of 25 mapped words for the case of en-de, helps it to achieve reasonable performance.
Revised: Table 4. BLISS(M) does not face this problem: adding supervision, even in the form of 25 mapped words for the case of en-de, helps it to achieve reasonable performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Mikolov et al.
Revised: Mikolov et al.

Original: (2013a)
Revised: (2013a)

Original: ﬁrst used anchor points to align two embedding spaces, leveraging the fact that these spaces exhibit similar structure across languages.
Revised: ﬁrst used anchor points to align two embedding spaces, leveraging the fact that these spaces exhibit similar structure across languages.

Original: Since then, several approaches have been proposed for learning bilingual dictionaries (Faruqui & Dyer, 2014; Zou et al., 2013; Xing et al., 2015).
Revised: Since then, several approaches have been proposed for learning bilingual dictionaries (Faruqui & Dyer, 2014; Zou et al., 2013; Xing et al., 2015).

Original: Xing et al.
Revised: Xing et al.

Original: (2015) showed that adding an orthogonal constraint signiﬁcantly improves performance, and admits a closed form solution.
Revised: (2015) showed that adding an orthogonal constraint signiﬁcantly improves performance, and admits a closed form solution.

Original: This was further corroborated by the work of Smith et al.
Revised: This was further corroborated by the work of Smith et al.

Original: (2017), who showed that in orthogonality was necessary for self-consistency.
Revised: (2017), who showed that in orthogonality was necessary for self-consistency.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2016) showed the equivalence between the different methods, and their subsequent work (Artetxe et al., 2018a) analyzed different techniques proposed in various works (like embedding centering, whitening etc.), and showed that leveraging a combination different methods showed signiﬁcant performance gains.
Revised: (2016) showed the equivalence between the different methods, and their subsequent work (Artetxe et al., 2018a) analyzed different techniques proposed in various works (like embedding centering, whitening etc.), and showed that leveraging a combination different methods showed signiﬁcant performance gains.

Original: However, the validity of this orthogonality assumption has of late come into question: Zhang et al.
Revised: However, the validity of this orthogonality assumption has of late come into question: Zhang et al.

Original: (2017a) found that the Wasserstein distance between distant language pairs was considerably higher , while Søgaard et al.
Revised: (2017a) found that the Wasserstein distance between distant language pairs was considerably higher , while Søgaard et al.

Original: (2018) explored the orthogonality assumption using eigenvector similarity.
Revised: (2018) explored the orthogonality assumption using eigenvector similarity.

Original: We ﬁnd our weak orthogonality constraint (along the lines of (Zhang et al., 2017b)) when used in our semi-supervised framework to be more robust to this.
Revised: We ﬁnd our weak orthogonality constraint (along the lines of (Zhang et al., 2017b)) when used in our semi-supervised framework to be more robust to this.

Original: There has also recently been an increasing focus on generating these bilingual mappings without an aligned bilingual dictionary, i.e., in an unsupervised manner.
Revised: There has also recently been an increasing focus on generating these bilingual mappings without an aligned bilingual dictionary, i.e., in an unsupervised manner.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2017b)
Revised: (2017b)

Original: and Lample et al.
Revised: and Lample et al.

Original: both use adversarial training for aligning two monolingual embedding spaces without any seed lexicon, while Zhang et al.
Revised: both use adversarial training for aligning two monolingual embedding spaces without any seed lexicon, while Zhang et al.

Original: (2017a) used a Wasserstein GAN to achieve this adversarial alignment, and use an earth-mover based ﬁne-tuning approach; while Grave et al.
Revised: (2017a) used a Wasserstein GAN to achieve this adversarial alignment, and use an earth-mover based ﬁne-tuning approach; while Grave et al.

Original: (2018) formulate this as a joint estimation of an orthogonal matrix and a permutation matrix.
Revised: (2018) formulate this as a joint estimation of an orthogonal matrix and a permutation matrix.

Original: However, we show that adding a little supervision, which is usually easy to obtain, improves performance.
Revised: However, we show that adding a little supervision, which is usually easy to obtain, improves performance.

Original: Another vein of research (Jawanpuria et al., 2018), (Artetxe et al., 2018b) has been to learn orthogonal mappings from both the source and the target embedding spaces into a common embedding space and doing the translations in the common embedding space.
Revised: Another vein of research (Jawanpuria et al., 2018), (Artetxe et al., 2018b) has been to learn orthogonal mappings from both the source and the target embedding spaces into a common embedding space and doing the translations in the common embedding space.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2017) and Søgaard et al.
Revised: (2017) and Søgaard et al.

Original: (2018) motivate the utility of using both the supervised seed dictionaries and, to some extent, the structure of the monolingual embedding spaces.
Revised: (2018) motivate the utility of using both the supervised seed dictionaries and, to some extent, the structure of the monolingual embedding spaces.

Original: They use iterative Procrustes reﬁnement starting with a small seed dictionary to learn a mapping; but doing may lead to sub-optimal performance for distant language pairs.
Revised: They use iterative Procrustes reﬁnement starting with a small seed dictionary to learn a mapping; but doing may lead to sub-optimal performance for distant language pairs.

Original: However, these methods are close to our methods in spirit, and consequently form the baselines for our experiments.
Revised: However, these methods are close to our methods in spirit, and consequently form the baselines for our experiments.

Original: Another avenue of research has been to try and modify the underlying embedding generation algorithms.
Revised: Another avenue of research has been to try and modify the underlying embedding generation algorithms.

Original: Cao et al.
Revised: Cao et al.

Original: (2016) modify the CBOW algorithm Mikolov et al.
Revised: (2016) modify the CBOW algorithm Mikolov et al.

Original: (2013b) by augmenting the
Revised: (2013b) by augmenting the

Original: CBOW loss to match the ﬁrst and second order moments from the source and target latent spaces, thereby ensuring the source and target embedding spaces follow the same distribution.
Revised: CBOW loss to match the ﬁrst and second order moments from the source and target latent spaces, thereby ensuring the source and target embedding spaces follow the same distribution.

Original: Luong et al.
Revised: Luong et al.

Original: (2015), in their work, use the aligned words to jointly learn the embedding spaces of both the source and target language, by trying to predict the context of a word in the other language, given an alignment.
Revised: (2015), in their work, use the aligned words to jointly learn the embedding spaces of both the source and target language, by trying to predict the context of a word in the other language, given an alignment.

Original: An issue with the proposed method is that it requires the retraining of embeddings, and cannot leverage a rich collection of precomputed vectors (like ones provided by Word2Vec (Mikolov et al., 2013b), Glove (Pennington et al., 2014) and FastText (Bojanowski et al., 2016)).
Revised: An issue with the proposed method is that it requires the retraining of embeddings, and cannot leverage a rich collection of precomputed vectors (like ones provided by Word2Vec (Mikolov et al., 2013b), Glove (Pennington et al., 2014) and FastText (Bojanowski et al., 2016)).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we analyze the validity of the orthogonality assumption and show that it breaks for distant language pairs.
Revised: In this work, we analyze the validity of the orthogonality assumption and show that it breaks for distant language pairs.

Original: We motivate the task of semi-supervised BLI by showing the shortcomings of purely supervised and unsupervised approaches.
Revised: We motivate the task of semi-supervised BLI by showing the shortcomings of purely supervised and unsupervised approaches.

Original: We ﬁnally propose a semi-supervised framework which combines the advantages of supervised and unsupervised approaches and uses a joint optimization loss to enforce a weak and ﬂexible orthogonality constraint.
Revised: We ﬁnally propose a semi-supervised framework which combines the advantages of supervised and unsupervised approaches and uses a joint optimization loss to enforce a weak and ﬂexible orthogonality constraint.

Original: We show that our framework obtains gains over several baseline models for numerous language pairs.
Revised: We show that our framework obtains gains over several baseline models for numerous language pairs.

Original: On analyzing the model errors, we ﬁnd that a large fraction of them arise due to polysemy and antonymy (An interested reader can ﬁnd the details in Appendix ( § 7.2).
Revised: On analyzing the model errors, we ﬁnd that a large fraction of them arise due to polysemy and antonymy (An interested reader can ﬁnd the details in Appendix ( § 7.2).

Original: An interesting line of future work would be to extend the method proposed here to account for polysemy in translation, possibly by leveraging the work of Upadhyay et al.
Revised: An interesting line of future work would be to extend the method proposed here to account for polysemy in translation, possibly by leveraging the work of Upadhyay et al.

Original: (2017), which uses multilingual context for sense disambiguation.
Revised: (2017), which uses multilingual context for sense disambiguation.

Original: Another confounding factor is synonyms and antonyms, which appear in similar contexts, and often incorrectly get translated to each other: leveraging the work done by Mrkˇsi´c et al.
Revised: Another confounding factor is synonyms and antonyms, which appear in similar contexts, and often incorrectly get translated to each other: leveraging the work done by Mrkˇsi´c et al.

Original: (2016) and Faruqui et al.
Revised: (2016) and Faruqui et al.

Original: (2014) might be an interesting way to mitigate this problem.
Revised: (2014) might be an interesting way to mitigate this problem.

Original:  A PPENDIX 7. T OY D ATASET EXPERIMENT
Revised:  A PPENDIX 7. T OY D ATASET EXPERIMENT

Original: [Figure]
Revised: [Figure]

Original: Unsupervised distribution matching solution remains invariant to the permutations of the words within a language.
Revised: Unsupervised distribution matching solution remains invariant to the permutations of the words within a language.

Original: One can easily construct a toy dataset on which this scenario can arise.
Revised: One can easily construct a toy dataset on which this scenario can arise.

Original: 4a shows a source and target distribution.
Revised: 4a shows a source and target distribution.

Original: Although, in the real dataset each point carries a different label, we only consider 2 labels for the sake of simplicity.
Revised: Although, in the real dataset each point carries a different label, we only consider 2 labels for the sake of simplicity.

Original: The correct transformation for matching distribution and labels is an anticlockwise rotation on the source.
Revised: The correct transformation for matching distribution and labels is an anticlockwise rotation on the source.

Original: Since the GAN does not see the labels, it just matches the distributions by half of the times either choosing the correct anticlockwise rotation or the incorrect clockwise rotation on source 4b.
Revised: Since the GAN does not see the labels, it just matches the distributions by half of the times either choosing the correct anticlockwise rotation or the incorrect clockwise rotation on source 4b.

Original: This problem can be solved by giving some labeled data correspondence and adding a supervised loss term.
Revised: This problem can be solved by giving some labeled data correspondence and adding a supervised loss term.

Original: 7. A NALYZING M ODEL E RRORS
Revised: 7. A NALYZING M ODEL E RRORS

Original: We characterize the mistakes made by the model, and ﬁnd that most fall into the following 4 categories:
Revised: We characterize the mistakes made by the model, and ﬁnd that most fall into the following 4 categories:

Original: Polysemy on the target side : These are the cases in which the predicted words and the gold translation are synonyms/hypernyms/hyponyms of each other.
Revised: Polysemy on the target side : These are the cases in which the predicted words and the gold translation are synonyms/hypernyms/hyponyms of each other.

Original: Polysemy on the source side : These are the cases in which the gold translations and the predicted words are different senses of the source word.
Revised: Polysemy on the source side : These are the cases in which the gold translations and the predicted words are different senses of the source word.

Original: Antonyms : The distribution of the context of antonyms is often very similar.
Revised: Antonyms : The distribution of the context of antonyms is often very similar.

Original: Unsurprisingly the word vectors of antonyms are quite similar.
Revised: Unsurprisingly the word vectors of antonyms are quite similar.

Original: This leads to cases where the predicted words and gold labels are antonyms of each other.
Revised: This leads to cases where the predicted words and gold labels are antonyms of each other.

Original: Words that occur in common contexts : Words that occur in numerous contexts often have poor word embeddings, since a single embedding can’t capture polysemy.
Revised: Words that occur in common contexts : Words that occur in numerous contexts often have poor word embeddings, since a single embedding can’t capture polysemy.

Original: Consequently, multiple such word embeddings that are frequent and have poor representations often get incorrectly translated to each other.
Revised: Consequently, multiple such word embeddings that are frequent and have poor representations often get incorrectly translated to each other.

Original: Some examples include proper nouns and numbers
Revised: Some examples include proper nouns and numbers

Original: We quantitatively estimate the fraction of errors due to these reasons using WordNet synsets.
Revised: We quantitatively estimate the fraction of errors due to these reasons using WordNet synsets.

Original: Given 2 synsets, WordNet provides a score denoting how similar two word senses are, based on the shortest path that connects the senses in the is-a (hypernym/hypnoym) taxonomy.
Revised: Given 2 synsets, WordNet provides a score denoting how similar two word senses are, based on the shortest path that connects the senses in the is-a (hypernym/hypnoym) taxonomy.

Original: The score is in the range 0 to 1.
Revised: The score is in the range 0 to 1.

Original: A score of 1 represents identity i.e. comparing a sense with itself will return 1.
Revised: A score of 1 represents identity i.e. comparing a sense with itself will return 1.

Original: We approximate the fraction of target polysemy errors by ﬁnding those cases for which the aforementioned similarity scores between the synsets of the predicted words and the gold translations ≥ 0 .
Revised: We approximate the fraction of target polysemy errors by ﬁnding those cases for which the aforementioned similarity scores between the synsets of the predicted words and the gold translations ≥ 0 .

Original: Similarly we approximate the fraction of source polysemy errors by ﬁnding those cases for which the similarity scores between the synsets of the source word and the predicted word ≥ 0 .
Revised: Similarly we approximate the fraction of source polysemy errors by ﬁnding those cases for which the similarity scores between the synsets of the source word and the predicted word ≥ 0 .

Original: Fig 4 shows these estimations for different language pairs.
Revised: Fig 4 shows these estimations for different language pairs.

Original: See Table 5a in ( § 7.2) for examples sampled from each of these error types.
Revised: See Table 5a in ( § 7.2) for examples sampled from each of these error types.

Original: Type of Error
Revised: Type of Error

Original: Source
Revised: Source

Original: Gold
Revised: Gold

Original: Predicted
Revised: Predicted

Original: Comments
Revised: Comments

Original: Target Polysemy
Revised: Target Polysemy

Original: Shadows 影 子 阴 影 synonyms
Revised: Shadows 影 子 阴 影 synonyms

Original: Target Polysemy
Revised: Target Polysemy

Original: Quest
Revised: Quest

Original: Quest
Revised: Quest

Original: Avventura synonyms
Revised: Avventura synonyms

Original: Source Polysemy
Revised: Source Polysemy

Original: Worn us´e vˆetement
Revised: Worn us´e vˆetement

Original: Gold: used, Predicted: cloth
Revised: Gold: used, Predicted: cloth

Original: Source Polysemy
Revised: Source Polysemy

Original: Bitter 苦 辛辣
Revised: Bitter 苦 辛辣

Original: Gold: bitter (taste), predicted: bitter (feeling)
Revised: Gold: bitter (taste), predicted: bitter (feeling)

Original: Antonyms
Revised: Antonyms

Original: Unofﬁcial
Revised: Unofﬁcial

Original: Ufﬁciale
Revised: Ufﬁciale

Original: Funzionario funzionario: ofﬁcial
Revised: Funzionario funzionario: ofﬁcial

Original: Antonyms
Revised: Antonyms

Original: Mature Mature
Revised: Mature Mature

Original: Jeune
Revised: Jeune

Original: Jeune: young
Revised: Jeune: young

Original: Antonyms
Revised: Antonyms

Original: Afraid
Revised: Afraid

Original: Paura
Revised: Paura

Original: Contento
Revised: Contento

Original: Gold: fear, Predicted: happy
Revised: Gold: fear, Predicted: happy

Original: Common Words
Revised: Common Words

Original: Everybody
Revised: Everybody

Original: Jeder
Revised: Jeder

Original: Spaß
Revised: Spaß

Original: Gold: Everybody,
Revised: Gold: Everybody,

Original: [Table]
Revised: [Table]

Original: 7. β ORTHOGONALITY PROJECTION VS . AUTOENCODING LOSS
Revised: 7. β ORTHOGONALITY PROJECTION VS . AUTOENCODING LOSS

Original: [Table]
Revised: [Table]

Original: Lample et al.
Revised: Lample et al.

Original: (2018) constraint the mapping matrix to be close to the manifold of orthogonal matrices by applying the following projection step after every update.
Revised: (2018) constraint the mapping matrix to be close to the manifold of orthogonal matrices by applying the following projection step after every update.

Original: [Equation]
Revised: [Equation]

Original: In our experiments we found out that the ﬁnal accuracy is highly sensitive to the value of the hyperparameter β
Revised: In our experiments we found out that the ﬁnal accuracy is highly sensitive to the value of the hyperparameter β

Original: (Table 6).
Revised: (Table 6).

Original: Our approach on the other hand uses an autoencoding loss which allows the model to ﬂexibly adjusts the degree of orthogonality in a data driven manner and works consistently well for one choice of the scaling of the autoencoding loss.
Revised: Our approach on the other hand uses an autoencoding loss which allows the model to ﬂexibly adjusts the degree of orthogonality in a data driven manner and works consistently well for one choice of the scaling of the autoencoding loss.

Original: 7. H YPER -P ARAMETERS
Revised: 7. H YPER -P ARAMETERS

Original: The following are the hyper parameters used in the experiments.
Revised: The following are the hyper parameters used in the experiments.

Original: The values separated by / are the different values tried in the parameter search.
Revised: The values separated by / are the different values tried in the parameter search.

Original: • Number of words per language considered for GAN training: top• Discriminator Parameters : – embedding dim:– hidden dim:– dropout prob: 0.1 (Only on the input layer) – label smoothing: 0.
Revised: • Number of words per language considered for GAN training: top• Discriminator Parameters : – embedding dim:– hidden dim:– dropout prob: 0.1 (Only on the input layer) – label smoothing: 0.

Original: • Generator Parameters • Initialization: Identity / Random Orthogonal • Mean Center: True • GAN Training Parameters – batch size:– Optimizer: SGD – Supervised loss optimizer: SGD / Adam – lr: 0.1 (with a schedule of 0.98 decay per round, and halved if CSLS metric does not improve over two rounds).
Revised: • Generator Parameters • Initialization: Identity / Random Orthogonal • Mean Center: True • GAN Training Parameters – batch size:– Optimizer: SGD – Supervised loss optimizer: SGD / Adam – lr: 0.1 (with a schedule of 0.98 decay per round, and halved if CSLS metric does not improve over two rounds).

Original: – Hubness Threshold:• f a = cosine
Revised: – Hubness Threshold:• f a = cosine

Revised:  P ERFORMANCE WITH DIFFERENT LEVELS OF SUPERVISION

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: RCSLS 22.

Revised: BLISS

Revised: [Table]

Revised: GeoMM 79.

Revised: RCSLS 80.

Revised: BLISS

Revised: [Table]