U NSUPERVISED H YPERALIGNMENT FOR M ULTILINGUAL W ORD E MBEDDINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the problem of aligning continuous word representations, learned in multiple languages, to a common space.
It was recently shown that, in the case of two languages, it is possible to learn such a mapping without supervision.
This paper extends this line of work to the problem of aligning multiple languages to a common space.
A solution is to independently map all languages to a pivot language.
Unfortunately, this degrades the quality of indirect word translation.
We thus propose a novel formulation that ensures composable mappings, leading to better alignments.
We evaluate our method by jointly aligning word vectors in eleven languages, showing consistent improvement with indirect mappings while maintaining competitive performance on direct word translation.
I NTRODUCTION
Pre-trained continuous representations of words are standard building blocks of many natural language processing and machine learning systems (Mikolov et al., 2013b).
Word vectors are designed to summarize and quantify semantic nuances through a few hundred coordinates.
Such representations are typically used in downstream tasks to improve generalization when the amount of data is scarce (Collobert et al., 2011).
The distributional information used to learn these word vectors derives from statistical properties of word co-occurrence found in large corpora (Deerwester et al., 1990).
Such corpora are, by design, monolingual (Mikolov et al., 2013b; Bojanowski et al., 2016), resulting in the independent learning of word embeddings for each language.
A limitation of these monolingual embeddings is that it is impossible to compare words across languages.
It is thus natural to try to combine all these word representations into a common multilingual space, where every language could be mapped.
Mikolov et al.
(2013a) observed that word vectors learned on different languages share a similar structure.
More precisely, two sets of pre-trained vectors in different languages can be aligned to some extent: a linear mapping between the two sets of embeddings is enough to produce decent word translations.
Recently, there has been an increasing interest in mapping these pre-trained vectors in a common space (Xing et al., 2015b; Artetxe et al., 2017), resulting in many publicly available embeddings in many languages mapped into a single common vector space (Smith et al., 2017; Conneau et al., 2017; Joulin et al., 2018).
The quality of these multilingual embeddings can be tested by composing mappings between languages and looking at the resulting translations.
As an example, learning a direct mapping between Italian and Portuguese leads to a word translation accuracy of 78 .
1% with a nearest neighbor (NN) criterion, while composing the mapping between Italian and English and Portuguese and English leads to a word translation accuracy of 70 . 7% only.
Practically speaking, it is not surprising to see such a degradation since these bilingual alignments are trained separately, without enforcing transitivity.
In this paper, we propose a novel approach to align multiple languages simultaneously in a common space in a way that enforces transitive translations.
Our method relies on constraining word translations to be coherent between languages when mapped to the common space.
Nakashole and Flauger (2017) has recently shown that similar constraints over a well chosen triplet of languages improve supervised bilingual alignment.
Our work extends their conclusions to the unsupervised case.
We show that our approach achieves competitive performance while enforcing composition.
P RELIMINARIES ON BILINGUAL ALIGNMENT
In this section, we provide a brief overview of bilingual alignment methods to learn a mapping between two sets of embeddings, and discuss their limits when used in multilingual settings.
2. S UPERVISED BILINGUAL ALIGNMENT
Mikolov et al.
(2013a) formulate the problem of word vector alignment as a quadratic problem.
Given two sets of n word vectors stacked in two n × d matrices X and Y and a n × n assignment matrix P ∈ { 0 , 1 } n × n built using a bilingual lexicon, the mapping matrix Q ∈ R
d × d is the solution of the least-square problem:
[Equation]
, which admits a closed form solution.
Restraining Q to the set of orthogonal matrices O d , improves the alignments (Xing et al., 2015b).
The resulting problem, known as Orthogonal Procrustes, still admits a closed form solution through a singular value decomposition (Schnemann, 1966). Alternative loss function. The (cid:96) 2 loss is intrinsically associated with the nearest neighbor (NN) criterion.
This criterion suffers from the existence of “hubs”, which are data points that are nearest neighbors to many other data points (Dinu et al., 2014). Alternative criterions have been suggested, such as the inverted softmax (Smith et al., 2017) and CSLS (Conneau et al., 2017). Recently, Joulin et al.
(2018) has shown that directly minimizing a loss inspired by the CSLS criterion signiﬁcantly improve the quality of the retrieved word translations. Their loss function, called RCSLS, is deﬁned as:
[Equation]
This loss is a tight convex relaxation of the CSLS cristerion for normalized word vectors, and can be efﬁciently minimized with a subgradient method. 2. U NSUPERVISED BILINGUAL A LIGNMENT : W ASSERSTEIN -P ROCRUSTES In the setting of unsupervised bilingual alignment, the assignment matrix P is unknown and must be learned jointly with the mapping Q . An assignment matrix represents a one-to-one correspondence between the two sets of words, i.e., is a bi-stochastic matrix with binary entries.
The set of assignment matrices P n is thus deﬁned as:
[Equation]
The resulting approach, called Wasserstein-Procrustes (Zhang et al., 2017a; Grave et al., 2018), jointly learns both matrices by solving the following problem:
[Equation]
This problem is not convex since neither of the sets P n and O d are convex.
Minimizing over each variable separately leads, however, to well understood optimization problems: when P is ﬁxed, minimizing over Q involves solving the orthogonal Procrustes problem.
When Q is ﬁxed, an optimal permutation matrix P can be obtained with the Hungarian algorithm.
A simple heuristic to address Eq.(2) is thus to use an alternate optimization. Both algorithms have a cubic complexity but on different quantities: Procrustes involves the dimension of the vectors, i.e., O ( d 3 ) (with d = 300 ), whereas the Hungarian algorithm involves the size of the sets, i.e., O ( n 3 ) (with n = 20 k– 200 k).
Directly applying the Hungarian algorithm is computationally prohibitive, but efﬁcient alternatives exist.
Cuturi (2013) shows that regularizing this problem with a negative entropy leads to a Sinkhorn algorithm, with a complexity of O ( n 2 ) up to logarithmic factors (Altschuler et al., 2017).
As for many non-convex problems, a good initial guess helps converge to better local minima.
Grave et al.
(2018) compute an initial P with a convex relaxation of the quadratic assignment problem.
We found, however, that the entropic regularization of the Gromov-Wasserstein (GW) problem (M´emoli, 2011) worked well in practice and was signiﬁcantly faster (Solomon et al., 2016;
Peyr´e et al., 2016):
[Equation]
The case ε = 0 corresponds to M ´ emoli’s initial proposal.
Optimizing the regularized version ( (cid:15) > 0 ) leads to a local minimum
ˆ P that can be used as an initialization to solve Eq.
Note that a similar formulation was recently used in the same context by Alvarez-Melis and Jaakkola (2018).
C OMPOSABLE M ULTILINGUAL A LIGNMENTS
In this section, we propose an unsupervised approach to jointly align N sets of vectors to a unique common space while preserving the quality of word translation between every pair of languages.
3. M ULTILINGUAL ALIGNMENT TO A COMMON SPACE
Given N sets of word vectors, we are interested in aligning these to a common target space T . For simplicity, we assume that this target space coincide with one of the word vector set.
The language associated with this vector set is denoted as “pivot” and is indexed by i = 0 .
A typical choice for the pivot, used in publicly available aligned vectors, is English (Smith et al., 2017; Conneau et al., 2017; Joulin et al., 2018). Aligning multiple languages to a common space consists in learning a mapping Q i for each language i , such that its vectors are aligned with the vectors of the pivot language up to a permutation matrix
P i :
[Equation]
This objective function decomposes over each language and does not guarantee good indirect word translation between pairs of languages that do not include the pivot.
A solution would be to directly enforce compositionality by adding constraints on the mappings. However, this would require to introduce mappings between all pairs of languages, leading to the estimation of O ( N 2 ) mappings simultaneously. Instead we leverage the fact that all the vector sets are mapped to a common space to enforce good alignments within this space.
With the convention that Q 0 is equal to the identity, this leads to the following problem:
[Equation]
where α ij > 0 weights the importance of the alignment quality between the languages i and j .
This formulation does not introduce any unnecessary mapping. It constrains all pairs of vector sets to be well aligned, instead of directly constraining the mappings.
Constraining the mappings would encourage coherence over the entire space, while we are interested in well aligned data; that is coherent mapping within the span of the word vectors.
Our approach takes its inspiration from the hyperalignment of multiple graphs (Goodall, 1991)
We refer to our approach as Unsupervised
Multilingual Hyperalignment
Choice of weights.
The weights on the alignments can be chosen to reﬂect prior knowledge about the relations between languages. For example, these weights can be a function of a rough language similarity measure extracted from the initial alignment.
It is not clear however how the weights should depend on these similarities: constraining similar languages with higher weights may be unnecessary, since they are already well aligned.
On the other hand, increasing weights of distant languages may lead to a problem where alignments are harder to learn.
In practice, we ﬁnd that simple weighting schemes work best with little assumptions.
For instance, uniform weights work well but tends to degrade the performance of translations to and from the pivot if the number of languages is large. Instead, we choose to use larger weights for direct alignments to the pivot, to insure that our multilingual objective does not degrade bilingual alignments to the pivot.
In practice, α ij is set to N if i or j is equal to 0 , and 1 otherwise.
Choice of loss function.
We consider the RCSLS loss of Joulin et al.
This loss is computationally expensive and wasteful to minimize it from scratch.
We thus optimize a (cid:96) 2 for the ﬁrst couple of epochs before switching to the RCSLS loss.
This two-step procedure shares some similarities with the use of a reﬁnement step after learning a ﬁrst rough alignment (Artetxe et al., 2017;
Conneau et al., 2017).
The (cid:96) 2 loss between two sets of normalized vectors is equivalent to a linear function, up to an additive constant C :
[Equation]
Where Q = Q i Q Tj if X and Y are the vectors from languages i and j .
We adapt the RCSLS loss of Eq. (1) to the unsupervised case by applying an assignment matrix to the target vectors: RCSLS
[Equation]
[Table]
Efﬁcient optimization.
Directly optimizing Eq. (4) is computationally prohibitive since N 2 terms are involved. We use a stochastic procedure where N pairs ( i, j ) of languages are sampled at each iteration and updated.
We sample pairs according to the weights α ij .
The RCSLS loss being slower to compute than the (cid:96) 2 loss, we ﬁrst optimize the latter for a couple of epochs before switching to the former.
The (cid:96) 2 loss is optimized with the same procedure as in Grave et al. (2018): alternate minimization with a Sinkhorn algorithm for the assignment matrix on small batches.
We then switch to a cruder optimization scheme when minimizing the RCSLS loss: we use a greedy assignment algorithm by taking the maximum per row.
We also subsample the number of elements to compute the K nearest neighbors from.
We restrict each set of vectors to its ﬁrst 20 k elements.
UMH runs on a CPU with 10 threads in less than 10 minutes for a pair of languages and in 2 hours for 6 languages.
R ELATED WORK
Bilingual word embedding alignment.
Since the work of Mikolov et al.
(2013a), many have proposed different approaches to align word vectors with different degrees of supervision, from fully supervised (Dinu et al., 2014; Xing et al., 2015a; Artetxe et al., 2016; Joulin et al., 2018) to little supervision (Smith et al., 2017; Artetxe et al., 2017) and even fully unsupervised (Zhang et al., 2017a; Conneau et al., 2017; Hoshen and Wolf, 2018).
Among unsupervised approaches, some have explicitly formulated this problem as a distribution matching: Cao et al.
(2016) align the ﬁrst two moments of the word vector distributions, assuming Gaussian distributions.
Others (Zhang et al., 2017b; Conneau et al., 2017) have used a Generative Adversarial Network framework (Goodfellow et al., 2014).
Zhang et al.
(2017a) shows that an earth mover distance can be used to reﬁne the alignment obtained from a generative adversarial network, drawing a connection between word embedding alignment and Optimal Transport (OT).
Artetxe et al.
(2018) proposes a stable algorithm to tackle distant pairs of languages and low quality embeddings.
Closer to our work, Grave et al.
(2018) and Alvarez-Melis and Jaakkola (2018) have proposed robust unsupervised bilingual alignment methods based on OT.
Our approach takes inspiration from their work and extend them to the multilingual setting.
Multilingual word embedding alignment.
Nakashole and Flauger (2017) showed that constraining coherent word alignments between triplets of nearby languages improves the quality of induced bilingual lexicons.
Jawanpuria et al.
(2018) recently showed similar results on any triplets of languages in the supervised case.
As opposed to our work, these approaches are restricted to triplets of languages and use supervision for both the lexicon and the choice of the pivot language.
Finally, independently of this work, Chen and Cardie (2018) has recently extended the bilingual method of Conneau et al. (2017) to the multilingual setting.
Optimal Transport.
Optimal transport (Villani, 2003; Santambrogio, 2015) provides a natural topology on shapes and discrete probability measures (Peyr´e et al., 2017), that can be leveraged with fast OT problem solvers (Cuturi, 2013; Altschuler et al., 2017).
Of particular interest is the
Gromov-Wasserstein distance (Gromov, 2007; M´emoli, 2011).
It has been used for shape matching under its primitive form (Bronstein et al., 2006; M´emoli, 2007) and under its entropy-regularized form (Solomon et al., 2016). We use the latter for our intialization.
Hyperalignment.
Hyperalignment, as introduced by Goodall (1991), is the method of aligning several shapes onto each other with supervision.
Recently, Lorbert and Ramadge (2012) extended this supervised approach to non-Euclidean distances.
We recommend Gower et al.
for a thorough survey of the different extensions of Procrustes and to Edelman et al.
(1998) for algorithms involving orthogonal constraints.
For unsupervised alignment of multiple shapes, Huang et al.
(2007) use a pointwise entropy based method and apply it to face alignment.
E XPERIMENTAL R ESULTS
Implementation Details.
We use normalized fastText word vectors trained on the Wikipedia Corpus (Bojanowski et al., 2016).
We use stochastic gradient descent (SGD) to minimize the RCSLS loss. We run the ﬁrst epoch with a batch size of 500 and then set it to 1 k. We set the learning rate to 0 .
1 for the (cid:96) 2 loss and to 25 for the RCSLS loss in the multilingual setting, and to 50 in the bilingual setting. For the ﬁrst two iterations, we learn the assignment with a regularized Sinkhorn. Then, for efﬁciency, we switch to a greedy assignment, by picking the max per row of the score matrix.
We initialize with the Gromov-Wasserstein approach applied to the ﬁrst 2 k vectors and a regularization parameter ε of 0 .
5 (Peyr ´ e et al., 2016).
We use the python optimal transport package.
Extended MUSE Benchmark.
We evaluate on the MUSE test datasets (Conneau et al., 2017), learning the alignments on the following 11 languages: Czech, Danish, Dutch, English, French,
German, Italian, Polish, Portuguese, Russian and Spanish.
MUSE bilingual lexicon are mostly translations to or from English.
For missing pairs of languages (e.g., Danish-German), we use the intersection of their translation to English to build a test set.
MUSE bilingual lexicon are built with an automatic translation system. The construction of the new bilingual lexicon is equivalent to translate with pivot.
Baselines.
We consider as baselines several bilingual alignment methods that are either supervised, i.e., Orthogonal Procrustes, GeoMM (Jawanpuria et al., 2018) and RCSLS (Joulin et al., 2018), or unsupervised, i.e., Adversarial (Conneau et al., 2017), ICP (Hoshen and Wolf, 2018),
Gromov-Wasserstein (GW) (Alvarez-Melis and Jaakkola, 2018) and Wasserstein Procrustes (“Wass
Proc.”)
(Grave et al., 2018).
We also compare with the unusupervised multilingual method of Chen and Cardie (2018).
All the unsupervised approaches, but GW, apply the reﬁnement step (ref.) of Conneau et al.
(2017) or of Chen and Cardie (2018).
5. T RIPLET ALIGNMENT
In these experiments, we evaluate the quality of our formulation in the simple case of language triplets.
One language acts as the pivot between the two others.
We evaluate both the direct translation to and from the pivot and the indirect translation between the two other languages.
An indirect translation is obtained by ﬁrst mapping the source language to the pivot, and then from the pivot to [Table] [Table] [Table]
ICP + ref. [Table] [Table] [Table]
W-Proc. + ref. [Table]
the target language. For completeness, we report direct translation between the source and target languages.
This experiment is inspired by the setting of Nakashole and Flauger ( Table 1 compares our approach trained on language pairs and triplets. We use a NN criterion to give insights on the quality of the dot product between mapped vectors. We test different settings: we change the pivot or the pair of languages, consider natural in-between and distant pivot.
We also consider languages harder to align to English, such as Finnish or Hungarian.
Overall, these variations have little impact on the performance. The direct translation to and from the pivot is not signiﬁcantly impacted by the presence or absence of a third language.
More interestingly, the indirect translation of a model trained with 3 languages often compares favorably with the direct translation from the source to the target. In comparison, the performance of indirect translation obtained with a bilingual model dropped by 6 − 8% .
This drop reduces to a couple of percents if a CSLS criterion is used instead of a NN criterion.
5. M ULTILINGUAL ALIGNMENT
In this set of experiments, we evaluate the quality of joint multilingual alignment on a larger set of 11 languages.
We look at the impact on direct and indirect alignments.
Direct word translation.
Table 2 shows a comparison of UMH with other unsupervised approaches on the MUSE benchmark. This benchmark consists of 5 translations to and from English.
In this experiment, UMH is jointly trained on 10 languages, plus English. The results on the remaining 5 languages are in the appendix.
We observe a slight improvement of performance
[Table]
of 0 .
3% compared to bilingual UMH, which is also consistent on the remaining 5 languages.
This improvement is not signiﬁcant but shows that our approach maintains good direct word translation.
Indirect word translation.
Table 3 shows the performance on indirect word translation with English as a pivot language.
We consider averaged accuracies among and across language families, i.e. Latin, Germanic and Slavic.
As expected, constraining the alignments signiﬁcantly improves over the bilingual baseline, by almost 4% . The biggest improvement comes from Slavic languages.
The smallest improvement is between Latin languages ( +2% ), since they are all already well aligned with English. In general, we observe that our approach helps the most for distant languages, but the relative improvements are similar across all languages. 5. A BLATION STUDY
In this section, we evaluate the impact of some of our design choices on the performance of UMH.
We focus in particular on the loss function, the weighting and the initialization. We also discuss the impact of the number of languages used for training on the performance of UMH. Impact of the loss function.
Table 2 compares bilingual UMH, with state-of-the-art unsupervised bilingual approaches on the MUSE benchmark.
All the approaches use the CSLS criterion. UMH directly learns a bilingual mapping with an approximation of the retrieval criterion. Bilingual UMH compares favorably with previous approaches ( +0 . 5% ). In particular, the comparison with “WProc.+ref” validates our choice of the RCSLS loss for UMH.
[Table] Impact of weights. Our choice of weights α ij favors direct translation over indirect translation. In this set of experiments, we look at the impact of this choice over simple uniform weights.
We compare UMH with uniform weights on direct and indirect word translation with a NN criterion. It is not surprising to see that uniform weights improve the quality of indirect word translation at the cost of poorer direct translation. In general, we experimentally found that our weighting makes UMH more robust when scaling to larger number of languages.
Impact of the initialization. In Sec. 2.2, we introduce a novel initialization based on the GromovWassertein approach. Instead, Grave et al. (2018) consider a convex relaxation of Eq.
(2) applied to centered vectors. Table 5 shows the impact of our initialization on the performance of UMH for direct and indirect bilingual alignment. We restrict this comparison to the 6 languages with existing [Table]
consider only direct translation to and and [Table] Impact of the number of languages. Table 6 shows the impact of the number of languages on
UMH. We train our models on 6 and 11 languages, and test them on the 6 common languages as in Chen and Cardie (2018). We use the same default hyper-parameters for all our experiments. These 6 languages are English, German, French, Spanish, Italian and Portuguese.
They are relatively simple to align together. Adding new and distant languages only affects the performance of UMH by less than a percent. This shows that UMH is robust to an increasing number of languages, even when these additional languages are quite distant from the 6 original ones. Finally, MAT+MPSR is slightly better than UMH ( +0 . 7% ) on indirect translation.
This difference is caused by our non uniform weights α ij that seems to have a stronger impact on small number of languages. Note that our approach is computationally more efﬁcient, training in 2h on a CPU instead of 5h on a GPU. Language tree. We compute a minimum spanning tree on the matrix of losses given by UMH between every pairs of languages, except English.
Three clusters appear: the Latin, Germanic and Slavic families chained as LatinGermanic-Slavic.
This qualitiative result is coherent with Table 3. However, the edges between languages make little sense, e.g., the edge between Spanish and Dutch. Our alignment based solely on embeddings is too coarse to learn subtle relations between languages.
C ONCLUSION
This paper introduces an unsupervised multilingual alignment method that maps every language into a common space while minimizing the impact on indirect word translation.
We show that a simple extension of a bilingual formulation signiﬁcantly reduces the drop of performance of indirect word translation. Our multilingual approach also matches the performance of previously published bilingual and multilingual approaches on direct translation. However, our current approach scales relatively well with the number of languages, but it is not clear if such a simple approach would be enough to jointly learn the alignment of hundreds of languages. A PPENDIX
We present detailed results of our experiments on 6 and 11 languages. en [Table] en
[Table]
unsupervised
[Table]
unconstrained
[Table]
[Table]
NN 80.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Bil.
[Table]
CSLS 82.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
unsupervised, constrained
Mul. - NN 79.
[Table]
[Table]
[Table]
[Table]
Mul.
[Table]
[Table]
[Table]
[Table]
unsupervised, unconstrained
Bil.
[Table]
NN
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Bil. - CSLS 83.
[Table]
[Table]
[Table]
[Table]
unsupervised, constrained Mul. - NN 80. [Table] [Table]
[Table]
[Table]
[Table]
[Table]
Mul. - CSLS
[Table]
[Table]
[Table]
[Table]
[Table]