A Relaxed Matching Procedure for Unsupervised BLI (2020)

Abstract: Recently unsupervised Bilingual Lexicon Induction(BLI) without any parallel corpus has attracted much research interest. One of the crucial parts in methods for the BLI task is the matching procedure. Previous works impose a too strong constraint on the matching and lead to many counterintuitive translation pairings. Thus We propose a relaxed matching procedure to find a more precise matching between two languages. We also find that aligning source and target language embedding space bidirectionally will bring significant improvement. We follow the previous iterative framework to conduct experiments. Results on standard benchmark demonstrate the effectiveness of our proposed method, which substantially outperforms previous unsupervised methods.

## introduction pretrained word embeddings @xcite are the basis of many other natural language processing and machine learning systems. word embeddings of a specific language contain rich syntax and semantic information. @xcite stated that the continuous embedding spaces exhibit similar structures across different languages, and we can exploit the similarity by a linear transformation from source embedding space to target embedding space. this similarity derives the bilingual lexicon induction(bli) task. the goal of bilingual lexicon induction is to align two languages' embedding space and generates word translation lexicon automatically. this fundamental problem in natural language processing benefits much other research such as sentence translation @xcite , unsupervised machine translation @xcite , cross-lingual information retrieval @xcite . recent endeavors (lample et al., 2018; alvarez-melis and jaakkola, 2018; grave et al., 2019; † yong zhang is the corresponding author. @xcite have proven that unsupervised bli's performance is even on par with the supervised methods. a crucial part of these approaches is the matching procedure, i.e., how to generate the translation plan. @xcite used gromov-wasserstein distance to approximate the matching between languages. @xcite regarded it as a classic optimal transport problem and used the sinkhorn algorithm @xcite to compute the translation plan. in this work, we follow the previous iterative framework but use a different matching procedure. previous iterative algorithms required to compute an approximate 1 to 1 matching every step. this 1 to 1 constraint brings out many redundant matchings. thus in order to avoid this problem, we relax the constraint and control the relaxation degree by adding two kl divergence regularization terms to the original loss function. this relaxation derives a more precise matching and significantly improves performance. then we propose a bidirectional optimization framework to optimize the mapping from source to target and from target to source simultaneously. in the section of experiments, we verify the effectiveness of our method, and results show our method outperforms many sota methods on the bli task. ## background the early works for the bli task require a parallel lexicon between languages. given two embedding matrices x and y with shape n × d (n:word number, d:vector dimension) of two languages and word x i in x is the translation of word y i in y , i.e., we get a parallel lexicon @xmath0. @xcite pointed out that we could exploit the similarities of monolingual embedding spaces by learning a linear transformation w such that under the unsupervised condition without parallel lexicon, i.e., vectors in x and y are totally out of order, @xcite proposed a domainadversarial approach for learning w . on account of the ground truth that monolingual embedding spaces of different languages keep similar spatial structures, alvarez-melis and jaakkola (2018) applied the gromov-wasserstein distance based on infrastructure to find the corresponding translation pairings between x and y and further derived the orthogonal mapping q. @xcite formulated the unsupervised bli task as we also take problem (2) as our loss function and our model shares a similar alternative framework with @xcite . however, we argue that the permutation matrix constraint on p is too strong, which leads to many inaccurate and redundant matchings between x and y , so we relax it by unbalanced optimal transport. @xcite extended the line of bli to the problem of aligning multiple languages to a common space. @xcite estimated q by a density matching method called normalizing flow. artetxe et al. (2018 ## proposed method in this section, we propose a method for the bli task. as mentioned in the background, we take problem (2) as our loss function and use a similar optimization framework in @xcite to estimate p and q alternatively. our method focuses on the estimation of p and tries to find a more precise matching p between xq and y . estimation of q is by stochastic gradient descent. we also propose a bidirectional optimization framework in section 3.2. ## experiments in this section, we evaluate our method in two settings. first, we conduct distillation experiments to verify the effectiveness of rmp and bidirectinal optimization. then we compare our method consisting of both rmp and bi-directional optimization with various sota methods on the bli task. datasets * we conduct word translation experiments on 6 pairs of languages and use pretrained word embedding from fasttext. we use the bilingual dictionaries opensourced in the work @xcite as our evaluate set.we use the csls retrieval method for evaluation as @xcite in both settings. all the translation accuracy reported is the precision at 1 with csls criterion. we open the source code on github † . ## conclusion this paper focuses on the matching procedure of bli task. our key insight is that the relaxed matching mitigates the counter-intuitive pairs by polysemy and obscure words, which is supported by comparing w.proc.-rmp with w.proc in