Cross-Lingual Word Embedding Refinement by \ell_{1} Norm Optimisation (2021)

Abstract: Cross-Lingual Word Embeddings (CLWEs) encode words from two or more languages in a shared high-dimensional space in which vectors representing words with similar meaning (regardless of language) are closely located. Existing methods for building high-quality CLWEs learn mappings that minimise the ℓ2 norm loss function. However, this optimisation objective has been demonstrated to be sensitive to outliers. Based on the more robust Manhattan norm (aka. ℓ1 norm) goodness-of-fit criterion, this paper proposes a simple post-processing step to improve CLWEs. An advantage of this approach is that it is fully agnostic to the training process of the original CLWEs and can therefore be applied widely. Extensive experiments are performed involving ten diverse languages and embeddings trained on different corpora. Evaluation results based on bilingual lexicon induction and cross-lingual transfer for natural language inference tasks show that the ℓ1 refinement substantially outperforms four state-of-the-art baselines in both supervised and unsupervised settings. It is therefore recommended that this strategy be adopted as a standard for CLWE methods.

## introduction cross-lingual word embedding (clwe) techniques have recently received significant attention as an effective means to support natural language processing applications for low-resource languages, e.g., machine translation @xcite and transfer learning @xcite . the most successful clwe models are the socalled projection-based methods, which learn mappings between monolingual word vectors with very little, or even zero, cross-lingual supervision @xcite @xcite . mainstream projection-based clwe models typically identify orthogonal mappings by * chenghua lin is the corresponding author. despite its popularity, work in various application domains has noted that ℓ 2 loss is not robust to noise and outliers. it is widely known in computer vision that ℓ 2 -loss-based solutions can severely exaggerate noise, leading to inaccurate estimates @xcite . in data mining, principal component analysis (pca) using ℓ 2 loss has been shown to be sensitive to the presence of outliers in the input data, degrading the quality of the feature space produced @xcite . previous studies have demonstrated that the processes used to construct monolingual and cross-lingual embeddings may introduce noise (e.g. via reconstruction error @xcite and structural variance @xcite ), making the presence of outliers more likely. empirical analysis of clwes also demonstrates that more distant word pairs (which are more likely to be outliers) have more influence on the behaviour of ℓ 2 loss than closer pairs. this raises the question of the appropriateness of ℓ 2 loss functions for clwes. compared to the conventional ℓ 2 loss, ℓ 1 loss (aka. manhattan distance) has been mathematically demonstrated to be less affected by outliers @xcite and empirically proven useful in computer vision and data mining @xcite @xcite . motivated by this insight, our paper proposes a simple yet effective post-processing technique to improve the quality of clwes: adjust the alignment of any cross-lingual vector space to minimise the ℓ 1 loss without violating the orthogonality constraint. specifically, given existing clwes, we bidirectionally retrieve bilingual vectors and optimise their manhattan distance using a numerical solver. the approach can be applied to any clwes, making the post-hoc refinement technique generic and applicable to a wide range of scenarios. we believe this to be the first application of ℓ 1 loss to the clwe problem. to demonstrate the effectiveness of our method, we select four state-of-the-art baselines and conduct comprehensive evaluations in both supervised and unsupervised settings. our experiments involve ten languages from diverse branches/families and embeddings trained on corpora of different domains. in addition to the standard bilingual lexicon induction (bli) benchmark, we also investigate a downstream task, namely cross-lingual transfer for natural language inference (nli). in all setups tested, our algorithm significantly improves the performance of strong baselines. finally, we provide an intuitive visualisation illustrating why ℓ 1 loss is more robust than it ℓ 2 counterpart when refining clwes (see fig. 1 ). our code is available at https://github.com/pzoom522/ l1-refinement . our contribution is three-fold: (1) we propose a robust refinement technique based on the ℓ 1 norm training objective, which can effectively enhance clwes; (2) our approach is generic and can be directly coupled with both supervised and unsupervised clwe models; (3) our ℓ 1 refinement algorithm achieves state-of-the-art performance for both bli and cross-lingual transfer for nli tasks. recent years have seen an increase in interest in projection-based methods, which train clwes by finding mappings between pretrained word vectors of different languages @xcite @xcite . since the input embeddings can be generated independently using monolingual corpora only, projection-based methods reduce the supervision required for training and offer a viable solution for low-resource scenarios. @xcite showed that the precision of the learned clwes can be improved by constraining the mapping function to be orthogonal, which is formalised as the so-called ℓ 2 orthogonal procrustes analysis (opa): while @xcite exploited an approximate and relatively slow gradient-based solver, more recent approaches such as @xcite and @xcite introduced an exact closed-form solution for eq. ( 1 ). originally proposed by @xcite , it utilises singular value decomposition (svd): while the aforementioned approaches still require some weak supervision (i.e., seed dictionaries), there have also been some successful attempts to train clwes in a completely unsupervised fashion. for instance, @xcite proposed a system called muse, which bootstraps clwes without any bilingual signal through adversarial learning. vecmap @xcite ) applied a self-learning strategy to iteratively compute the optimal mapping and then retrieve bilingual dictionary. comparing muse and vecmap, the latter tends to be more robust as its similarity-matrixbased heuristic initialisation is more stable in most cases @xcite . very recently, some studies bootstrapped unsupervised clwes by jointly training word embeddings on concatenated corpora of different languages and achieved good performance @xcite . the ℓ 2 refinement algorithm. clwe models often apply ℓ 2 refinement, a post-processing step shown to improve the quality of the initial alignment (see @xcite for survey). given existing clwes {x la , x lb } for languages l a and l b , bidirectionally one can use approaches such as the classic nearest-neighbour algorithm, the inverted softmax @xcite and the crossdomain similarity local scaling (csls) @xcite to retrieve two bilingual dictionaries d @xmath0. note that word pairs in d @xmath1, as they form "mutual translations". next, one can compose bilingual embedding matrices a and b by aligning word vectors (rows) using the above word pairs. finally, a new orthogonal mapping is learned to fit a and b based on least-square regressions, i.e., perform ℓ 2 opa described in eq. ( 1 ). early applications of ℓ 2 refinement applied a single iteration, e.g. @xcite . due to the wide adoption of the closed-form ℓ 2 opa solution (cf. eq. ( 2 )), recent methods perform multiple iterations. the iterative ℓ 2 refinement strategy is an important component of approaches that bootstrap from small or null training lexicons @xcite . however, a single step of refinement is often sufficient to create suitable clwes @xcite . ## methodology a common characteristic of clwe methods that apply the orthogonality constraint is that they optimise using ℓ 2 loss (see § 2). however, outliers have disproportionate influence in ℓ 2 since the penalty increases quadratically and this can be particularly problematic with noisy data since the solution can "shift" towards them @xcite . the noise and outliers present in real-world word embeddings may affect the performance of ℓ 2 -lossbased clwes. the ℓ 1 norm cost function is more robust than ℓ 2 loss as it is less affected by outliers @xcite . therefore, we propose a refinement algorithm for improving the quality of clwes based on ℓ 1 loss. this novel method, which we refer to as ℓ 1 refinement, is generic and can be applied post-hoc to improve the output of existing clwe models. to our knowledge, the use of alternatives to ℓ 2 -loss-based optimisation has never been explored by the clwe community. to begin with, analogous to ℓ 2 opa (cf. eq. ( 1 )), ℓ 1 opa can be formally defined and rewritten as larger values for α lead to closer approximations to sgn(•) but reduce the smoothing effect. this approach has been used in many applications, such as the activation function of long short-term memory networks @xcite . however, in practice, we find that eq. ( 4 ) remains unsolvable in our case with standard gradient-based frameworks for two reasons. first, α has to be sufficiently large in order to achieve a good approximation of sgn(•). otherwise, relatively small residuals will be down-weighted during fitting and the objective will become biased towards outliers, just similar to ℓ 2 loss. however, satisfying this requirement (i.e., large α) will lead to the activation function tanh(α(•)) becoming easily saturated, resulting in an optimisation process that becomes trapped during the early stages. in other words, the optimisation can only reach an unsatisfactory local optimum. second, the orthogonality constraint (i.e., @xmath2. we address these challenges by adopting the approaches proposed by @xcite . this method explicitly encourages the solver to only explore the desired manifold o thereby reducing the ℓ 1 solver's search space and difficulty of the optimisation problem. we begin by calculating the gradient ∇ w.r.t. the objective in eq. ( 4 ) through matrix differentiation: here i is an identity matrix with the shape of m. with eq. ( 6 ) defining the optimisation flow, our ℓ 1 loss minimisation problem reduces to an integration problem, as previous work has demonstrated that applying the ℓ 1 -loss-based algorithms from a good initial state can speed up the optimisation. for instance, @xcite found that feature spaces created by ℓ 2 pca were severely affected by noise. replacing the cost function with ℓ 1 loss significantly reduced this problem, but required expensive linear programming. to reduce the convergence time, brooks and jot (2013) exploited the first principal component from the ℓ 2 solution as an initial guess. similarly, when reconstructing corrupted pixel matrices, ℓ 2 -loss-based results are far from satisfactory; using ℓ 1 norm estimators can improve the quality, but are too slow to handle large-scale datasets @xcite . however, taking the ℓ 2 optima as the starting point allowed less biased reconstructions to be learned in an acceptable time @xcite . inspired by these works, we make use of ℓ 1 refinement to carry out post-hoc enhancement of existing clwes. our full pipeline is described in algorithm 1 ℓ 1 refinement input: clwes {xl a , xl b } output: updated clwes {xl a m , xl b } 1: dl @xmath3: dl @xmath4: d ← dl @xmath5: a, b ← looks up for d in xl a , xl b 5: perform integration to solve eq. ( 7 ) for m , with initial value m0 ← i, until stopping criteria are met algorithm 1 (see § 4.3 for implemented configurations). in common with ℓ 2 refinement (cf. § 2), steps 1-4 bootstrap a synthetic dictionary d and compose bilingual word vector matrices a and b which have reliable row-wise correspondence. taking them as the starting state, in step 5 an identity matrix naturally serves as our initial solution m 0 . during the execution of eq. ( 7 ), we record ℓ 1 loss per iteration and see if either of the following two stopping criteria have been satisfied: (1) the updated ℓ 1 loss exceeds that of the previous iteration; ## results we evaluate the effectiveness of the proposed ℓ 1 refinement technique on two benchmarks: bilingual lexicon induction (bli), the de facto standard for measuring the quality of clwes, and a downstream natural language inference task based on cross-lingual transfer. in addition to comparison against state-of-the-art clwe models, we also report the performance of the single-iteration ℓ 2 refinement method which follows steps 1-4 of algorithm 1 then minimises ℓ 2 loss in the final step. to reduce randomness, we executed each model in each setup three times and the average accuracy (acc, aka. precision at rank 1) is reported. following @xcite , by comparing scores achieved before and after ℓ 1 refinement, statistical significance is indicated via the p-value of two-tailed t-tests with bonferroni correction @xcite (note that p-values are not recorded for tab. 2b given the small number of runs). ## conclusion and future work this paper proposes a generic post-processing technique to enhance clwe performance based on optimising ℓ 1 loss. this algorithm is motivated by successful applications in other research fields (e.g. computer vision and data mining) which exploit the ℓ 1 norm cost function since it has been shown to be more robust to noisy data than the commonly-adopted ℓ 2 loss. the approach was evaluated using ten diverse languages and word embeddings from different domains on the popular bli benchmark, as well as a downstream task of cross-lingual transfer for nli. results demonstrated that our algorithm can significantly improve the quality of clwes in both supervised and unsupervised setups. it is therefore recommended that this straightforward technique be applied to improve performance of clwes. the convergence speed of the optimiser prevented us from performing ℓ 1 loss optimisation over multiple iterations. future work will focus on improving the efficiency of our ℓ 1 opa solver, as well as exploring the application of other robust loss functions within clwe training strategies.