Semi-Supervised Bilingual Lexicon Induction with Two-way Interaction (2020)

Abstract: Semi-supervision is a promising paradigm for Bilingual Lexicon Induction (BLI) with limited annotations. However, previous semisupervised methods do not fully utilize the knowledge hidden in annotated and nonannotated data, which hinders further improvement of their performance. In this paper, we propose a new semi-supervised BLI framework to encourage the interaction between the supervised signal and unsupervised alignment. We design two message-passing mechanisms to transfer knowledge between annotated and non-annotated data, named prior optimal transport and bi-directional lexicon update respectively. Then, we perform semi-supervised learning based on a cyclic or a parallel parameter feeding routine to update our models. Our framework is a general framework that can incorporate any supervised and unsupervised BLI methods based on optimal transport. Experimental results on MUSE and VecMap datasets show significant improvement of our models. Ablation study also proves that the two-way interaction between the supervised signal and unsupervised alignment accounts for the gain of the overall performance. Results on distant language pairs further illustrate the advantage and robustness of our proposed method.

## introduction bilingual lexicon induction (bli) is of huge interest to the research frontier. bli methods learn cross-lingual word embeddings from separately trained monolingual embeddings. bli is believed to be a promising way to transfer semantic information between different languages, and spawns lots of nlp applications like machine translation @xcite , part of speech (pos) tagging @xcite , parsing † yong zhang is the corresponding author. @xcite , and document classification @xcite . the key step of bli is to learn a transformation between monolingual word embedding spaces @xcite , which could be further used for translation retrieval or cross-lingual analogy tasks. however, it is hard to obtain the high quality transformation with low supervision signals, i.e. with limited annotated lexicon. thus, some semi-supervised bli methods @xcite are proposed to make use of annotated and non-annotated data. @xcite bootstrapped the supervised lexicon to enhance the supervision but ignored the knowledge in non-annotated data. meanwhile, @xcite combined the unsupervised bli loss that captured the structural similarity in word embeddings @xcite with the supervised loss @xcite . however, this loss combination still performed poorly since the bad supervised optimization under limited annotations, see the experiment part for details. as a result, existing semisupervised bli methods suffer from low effectiveness @xcite or low robustness @xcite . in this work, we focus on designing a new semisupervised bli method to make full use of both annotated and non-annotated data. we propose a novel framework with two different strategies, which exceeds the previous separate @xcite semi-supervised methods by emphasizing the two-way interaction between the supervised signal and unsupervised alignment. in this framework, supervised training tries to align the parallel lexicon and unsupervised training can exploit the structure similarity between monolingual embedding spaces. the foundation of twoway interaction is in two carefully designed message passing mechanisms, see section 3.1 and 3.2. two-way interaction enables semi-supervised bli to guide the exploitation of structural similarity by unsupervised procedure @xcite and extend insufficient lexicon for supervised procedure @xcite simultaneously, see figure 1 . in this paper, we only consider the unsupervised bli methods based on optimal transport (ot) @xcite @xcite , which have achieved impressive results on bli task. more specifically, the contributions of this paper are listed below. pot enables the ot-based unsupervised bli approach to be guided by any prior bli transformation, i.e. transfers what is learned by supervised bli method to the unsupervised bli method. blu employs the alignment results in bi-directional retrieval to enlarge the annotated data, and thus enhances the supervised training by unsupervised bli transformation. ## background in this section, we describe the basic formulation of related supervised and unsupervised bli methods. we define two embedding matrices x, @xmath0, where n is the number of words and d is the dimension of the word embedding. the key to supervised bli is the parallel lexi-con between two languages, say word x i in x is translated to word y i in y . @xcite suggested regarding supervised bli as a regression problem aligning word embeddings by a linear transformation q . ## message passing in bli in this section, we present two message passing mechanisms for semi-supervised framework, including pot and blu. pot is proposed to enhance the unsupervised bli by the knowledge passed from the supervised bli. meanwhile blu enhances the supervised bli by the additional lexicon based on the unsupervised retrieval results. therefore, pot and blu form the two-way interaction between the supervised signal and unsupervised alignment. ## semi-supervision with two-way interaction in the previous section, we have presented two message passing mechanisms to enhance supervised bli and ot-based unsupervised bli by prior transformation q unsup and q sup , respectively. moreover, recent state-of-the-art (sota) supervised(sup) and unsupervised(u nsup) approaches are all based on stochastic optimization rather than the closed-form solution. this means that all sota sup and u nsup approaches can be considered as a module that operates on the feed-in parameter q. therefore, we propose two different strategies for semi-supervision that emphasize the two-way interaction between the supervised signal and unsupervised alignment based on the message passing mechanisms, see figure 1 . all sota sup and ot-based u nsup methods can be plugged into the proposed framework seamlessly. ## experiment in this section, we conduct extensive experiments to evaluate the performance of css and pss. we open the source code on github * . ## conclusions in this paper, we introduce the two-way interaction between the supervised signal and unsupervised alignment by proposed pot and blu message passing mechanisms. pot guides the ot-based unsupervised bli by prior bli transformation. blu employs a bidirectional retrieval to enlarge the annotated data and stabilize the training of supervised bli approaches. ablation study shows that the two-way interaction by pot and blu is the key to significant improvement. based on the message passing mechanisms, we design two strategies of semi-supervised bli to integrate supervised and unsupervised approaches, css and pss, which are constructed on cyclic and parallel strategies respectively. the results show that css and pss achieve sota results over two popular datasets. as css and pss are compatible with any supervised bli and ot-based unsupervised bli approaches, they can also be applied to the latent space optimization.