Cross-lingual Feature Extraction from Monolingual Corpora for Low-resource Unsupervised Bilingual Lexicon Induction (2022)

Abstract: Despite their progress in high-resource language settings, unsupervised bilingual lexicon induction (UBLI) models often fail on corpora with low-resource distant language pairs due to insufficient initialization. In this work, we propose a cross-lingual feature extraction (CFE) method to learn the cross-lingual features from monolingual corpora for low-resource UBLI, enabling representations of words with the same meaning leveraged by the initialization step. By integrating cross-lingual representations with pre-trained word embeddings in a fully unsupervised initialization on UBLI, the proposed method outperforms existing state-of-the-art methods on low-resource language pairs (EN-VI, EN-TH, EN-ZH, EN-JA). The ablation study also proves that the learned cross-lingual features can enhance the representational ability and robustness of the existing embedding model.

## introduction bilingual lexicon induction (bli) has aroused great interest in the nlp research frontier. bli aims to induce word translation pairs by aligning word embeddings trained independently from monolingual corpora. bli has contributed to many nlp tasks, including unsupervised machine translation @xcite , cross-lingual dependency parsing @xcite and cross-lingual information retrieval. unsupervised bli has achieved reasonable results compared with semi-supervised works in highresource language settings, in which adversarial training were used in @xcite ; @xcite . these methods focused their attention on every single word, thus ignoring the relevance between words. @xcite proposed a method (vecmap) using a similarity matrix as an initial solution to learn the second-order structural similarity of the embeddings. another study directly leveraged an aligned similarity matrix instead of using the embedding matrix @xcite . recently, @xcite proposed a robust refinement technique based on the ℓ 1 norm training objective. the methods above learned bilingual spaces by orthogonally projecting one monolingual space to another. yet, evidence suggests that monolingual spaces, especially those of etymologically and typologically distant languages, are far from isomorphic @xcite @xcite . @xcite relaxed the orthogonality constraint to improve the performance of bli further. @xcite depicted a non-linear method using an encoder and decoder to learn the mapping in the latent space. @xcite proposed a joint training method using word alignments from parallel corpora as the supervision signals to align multilingual contextualized representations. while the methods mentioned above can leverage pre-trained embeddings in bli, they lack the means to incorporate richer information like cross-lingual features from monolingual data. when it comes to low-resource and non-cognate language, the characterization capability of pretrained embeddings is limited, which leads to the degeneration of these models. on another strand of work, traditional works for bli used the statistical methods to search the crosslingual signals @xcite @xcite @xcite @xcite . in recent years, e and zhou (2022) proposed a more robust method. they formally defined the semantic embedding of words in a mathematical way instead of machine learning. however, their method is limited to non-topic words and requires high quality monolingual data to achieve good results. the methods mentioned above have been reliant on a high-resource language condition. regarding low-resource and non-cognate language pairs, the characterization capability of pre-trained embeddings is limited, as only short-distance dependencies are available. in this paper, we propose a novel unsupervised method for bli based on crosslingual feature extraction. furthermore, we design two ways to integrate the cross-lingual features with the pre-trained embeddings, which show complementary effects according to the experimental results. in summary, this paper makes the following contributions: we expect the words with the same meaning in different languages have the similar representations and we can use it to initialize the ubli directly. ## background in this section, we describe the basic formulation of related supervised and unsupervised bli methods. let x, @xmath0, where n is the number of words and d is the dimension of the word embedding. the key to supervised bli is the parallel lexicon between two languages. let x * , y * ∈ r k×d represent parallel embedding matrices, say @xcite pointed out that a linear transformation w * could be used to map two monolingual embeddings to a shared space. @xcite solved problem (1) by adding an orthogonal constraint on w . therefore, there is a closed-form solution to this problem called procrutes: @xmath1, where u and v are defined by the svd decomposition of y ⊤ x. for unsupervised bli, embedding matrices x and y are totally out of order. therefore, unsupervised bli needs a permutation matrix @xmath2, 1} n×n to shuffles the row of y : problem (2) can be solved by minimizing w and p in an iterative way. @xcite proposed a stochastic algorithm to initialize w and p randomly and estimate them in a joint way. however, effectively minimizing p is hard. the key to unsupervised bli is how to solve p approximately. @xcite proposed an adversarial method to initialize the initial dictionary. @xcite has shown that two equivalent words in different languages have a similar distribution. therefore, they initialized matrix p based on similarity matrices of monolingual embeddings m @xmath3. they used the secondorder similarity of pre-trained embeddings to obtain a better initial dictionary. then they applied a self-learning strategy to iteratively compute the optimal mapping and retrieve bilingual dictionary until convergence. very recently, @xcite proposed a method to jointly train word embeddings on concatenated corpora of different languages and achieved good results. in summary, the foundation for bli is the parallel lexicon for initialization, especially for unsupervised bli. therefore, a high-quality initialization is the key for unsupervised bli. ## methodology in this section, we propose a novel framework to solve the problems of ubli in low-resource scenarios. first and foremost, the cfe is used to extract cross-lingual representations from monolingual data. then the ecb and the scb are used to integrate the cross-lingual features with pre-trained embeddings, either by concatenation or similarity weighting. ## experiment in this section, we report the results obtained with our method. we first evaluate the dataset we trained in section 5.1. second, we present our main results in section 5.2, thirdly we test the performance our cross-lingual feature in section 5.3, then we do ablation tests in section 5.4 to measure the contribution of each component and finally we compare the different initialization methods (3.2) in section 5.5. ## related work unsupervised bilingual lexicon induction (ubli) is an important task of machine translation. the existing methods for unsupervised bilingual lexicon induction are divided into two directions. the first is based on statistic method, and the other is based on the pre-trained embeddings. most of the methods follow the same procedure that is to find an initial solutions and then learning a mapping method between two embedding spaces. the key of these methods is finding an initial solution. for statistical methods, @xcite induced translations for words by using a generative model based on canonical correlation analysis, which explains the monolingual lexicons in terms of latent matchings. @xcite proposed a bilingual latent dirichlet allocation model for finding translations of terms in comparable corpora without using any linguistic resources. e and zhou (2022) proposed a method in a more mathematical way. their markov semantic model characterized the meaning of words with language-independent numerical fingerprints. in recent years, most methods initialized seed dictionary based on pre-trained embeddings. these methods can be divided into three categories. the first category is using adversarial methods @xcite @xcite . they trained a generator to find a mapping between two embedding spaces and a discriminator to distinguish the mapped source embedding from the target embedding. the second category is based on the structure of embedding space. @xcite showed the fact that two equivalent words in different languages should have a similar distribution, and used the secondorder similarity of pre-trained embeddings as an initialization of ubli. the third category is based on a non-linear mapping method. @xcite removed the orthogonal constraint of the mapping method. @xcite proposed a non-linear mapping in the latent space of two independently pre-trained autoencoders. all these methods only use one kind of feature. different from their methods, we leverage both monolingual corpus and word-level pre-trained embeddings to get richer information and achieve better accuracy. ## conclusion in this paper, we propose a method to extract crosslingual features through monolingual corpora, combined with pre-trained embeddings in two kinds to initial ubli. the experiments show that our method outperforms existing state-of-the-art methods on low-resource language pairs (en-vi, en-th, en-zh, en-ja). the ablation study demonstrates that the induced cross-lingual features have a complementary effect to pre-trained embeddings. besides, we also offer a muse-equivalent dataset with monolingual corpora. in the future, we will develop a more robust way of extracting cross-lingual features for lexicon induction. extending ubli to the phrase level is also a topic of interest.