A Simple and Effective Approach to Robust Unsupervised Bilingual Dictionary Induction (2020)

Abstract: Unsupervised Bilingual Dictionary Induction methods based on the initialization and the self-learning have achieved great success in similar language pairs, e.g., English-Spanish. But they still fail and have an accuracy of 0% in many distant language pairs, e.g., English-Japanese. In this work, we show that this failure results from the gap between the actual initialization performance and the minimum initialization performance for the self-learning to succeed. We propose Iterative Dimension Reduction to bridge this gap. Our experiments show that this simple method does not hamper the performance of similar language pairs and achieves an accuracy of 13.64 55.53% between English and four distant languages, i.e., Chinese, Japanese, Vietnamese and Thai.

## introduction unsupervised bilingual dictionary induction (ubdi) is a task that aims to find the word translations given the monolingual word embeddings of two languages. recent ubdi methods have shown promising results on similar language pairs such as english-spanish @xcite @xcite @xcite . these methods are mostly based on the initialization and the self-learning. the initialization first constructs a dictionary from the word embeddings, then the self-learning starts with this dictionary and alternates between refining the source-target word embedding mapping and inducing a new dictionary with this mapping. despite the success of ubdi, recent work has questioned the robustness of ubdi methods on distant language pairs @xcite @xcite , e.g., english-japanese. they show that even for the most robust system vecmap @xcite , it still fails and has an accuracy of 0% on 87 out of 210 distant language pairs @xcite . to be consistent with @xcite , we define a system 'succeeds' when it has an accuracy above 5% and 'fails' otherwise. previous work has investigated how different properties of languages have an impact on ubdi performance @xcite . in this paper, we take a step further to inspect which part of vecmap breaks down. with a novel similarity metric to evaluate the initialization performance, we observe a gap between the actual initialization performance and the minimum initialization performance for the self-learning to succeed in distant language pairs. we find that the dimension reduction approach is very effective in bridging this gap. therefore, we propose iterative dimension reduction (idr) to improve the robustness of vecmap and avoid performance loss due to dimension reduction. idr first reduces the dimension of word embeddings and performs unsupervised learning on them. then it initializes the self-learning on larger dimension embeddings using this learned system. this simple dimension reduction removes unimportant or noisy features, making the algorithm easier to find a proper solution to distant language pairs. entry en-zh zh-en en-ja ja-en en-vi vi-en en-th th-en vecmap accuracy [%] 0.07 0 1.03 32.67 0.73 0.73 0 0.07 maximum accuracy [%] 37.07 35.20 49.01 33.36 47.13 57.80 24.20 17.76 minimum dictionary size 150 74 74 97 137 110 147 154 we evaluate our approach on four similar european language pairs, including english-{spanish, french, italian, german} (en-{es, fr, it, de}), and four distant language pairs, including english-{chinese, japanese, vietnamese, thai} (en-{zh, ja, vi, th}). our method not only has a close performance to the vecmap baseline in similar language pairs but also succeeds in all distant language pairs. in four distant language pairs, our method has an accuracy of 13.64∼55.53%, whereas the vecmap baseline has an accuracy of 0% in most cases, as shown in the first row of for the current dictionary d, where d @xmath0. an optimal solution is given by w @xmath1, where u sv @xmath2; -a new dictionary d is induced by using the csls retrieval @xcite to extract nearest neighbors in the similarity matrix @xmath3. to avoid being trapped in a poor local optimum and encourage the exploration of possible word translations, similarity scores in p are kept with a probability p and set to 0 otherwise. there are some pre-processing and post-processing steps that are crucial to vecmap: since vecmap is pipelined by the initialization and the self-learning, we can assume the failure of unsupervised learning comes from either or both of these two components. two hypotheses arise: it is easy to verify the first hypothesis: we start the self-learning with a human-annotated seed dictionary. this way assumes a perfect initialization and thus eliminates the impact of the initialization. two natural questions come from the second hypothesis: one might expect the accuracy is a good proxy of the quality of a dictionary. but the accuracy only takes the correct translations into account. the intuition is that though the system fails to find the correct translation, it can still be useful if its translations are close to the correct answers. thus we evaluate the average cosine similarity between the word embeddings of the system translations and the correct answers, dubbed dictionary similarity. it will score high if the translations are close to the correct answers. figure 1 shows how the self-learning performs when starting from dictionaries with different dictionary similarities. these dictionaries are constructed by randomly replacing the translations in the initial dictionary. we can see that the self-learning only succeeds when the dictionary similarities of these starting dictionaries are above some thresholds. these thresholds represent the minimum similarity that the initial dictionary should have for the self-learning to succeed. we test the dictionary similarity of the initial dictionary. as shown in figure 2 , the actual initial dictionary similarities of the initialization are above the thresholds in similar language pairs, but it is the opposite in distant language pairs. the gap between the actual initialization similarity and the minimum similarity for the self-learning to succeed implies the failure of vecmap in distant language pairs. ## proposed method as the gap is determined by embeddings and the algorithm, one can improve the algorithm to bridge this gap. in this work, we start with a different angle by simplifying embeddings to make the algorithm easier to succeed. concretely, we run the dimension reduction on the word embeddings. in principle, the dimension reduction algorithm will drop features that are less important in explaining the data. it can be considered as a way to remove the noise and clean up the data. here we choose principal component analysis (pca) @xcite for study. spanish japanese pca first computes the covariance matrix of the features in the embeddings. this matrix represents the correlation among features. then pca performs the eigenvalue decomposition on this covariance matrix to obtain a set of eigenvectors. finally, pca uses eigenvectors with the highest n eigenvalues to project the raw embeddings to space with a lower dimension n. however, this simple application of pca only works in a few languages. we find that the highest eigenvalue is much larger than the others in most of the failed languages, e.g., it explains 96% variance in japanese while 5.6% in spanish as shown in the first row of ## analysis 6.1 ablation study on the other hand, the dropmax trick is crucial to en-{zh, ja} and their reverse. this fact relates well with the observation in figure 5 : the hubness level before and after applying the dropmax trick. ## related work in recent years a number of methods have been proposed to learn bilingual dictionary from monolingual word embeddings. early work @xcite relies on a seed dictionary to learn the source-target word embedding mapping. @xcite enforce the word embeddings to be of unit length and the orthogonal constraint on the linear mapping. @xcite on the other hand use canonical correlation analysis (cca) to project both source and target embeddings to a common low-dimensional space. @xcite show that the above methods are variants of the same objective. @xcite further show that this objective is closely related to the orthogonal procrustes problem. @xcite obtain competitive results using the self-learning with a seed dictionary of only 25 word pairs. adversarial methods. @xcite attempt the unsupervised bilingual dictionary induction task using the adversarial network. they use a generator to transform the source word embeddings to the target word embeddings and a discriminator to classify whether the given embedding is sampled from the true target word embeddings or generated by the generator. the generator is trained to fool the discriminator and the discriminator is trained to identify the generated word embeddings. in the end, the generator will be used to induce the bilingual dictionary. their following work @xcite minimizes earth-mover's distance between the transformed source and target embeddings distribution. @xcite improve the results by treating the dictionary produced by the adversarial network as the seed dictionary of the self-learning. to mitigate the hubness problem @xcite , they propose an effective nearest neighbors retrieval method csls for dictionary induction. @xcite minimize sinkhorn distance instead and introduce the circle consistency such that a source word embedding can be translated back after translating it to a target word. @xcite extract latent codes from word embeddings and align words according to their latent codes. ## conclusion in this work, we pinpoint in which part the representative ubdi system, vecmap, fails on distant language pairs. we identify a gap between the initialization performance and the minimum initialization performance for the self-learning to succeed, which is responsible for its failure. we propose iterative dimension reduction to bridge this gap. our method obtains substantial gains in distant language pairs without scarifying the performance of similar language pairs. it has shown to robust to the four distant language pairs we experiment with.