Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment (2021)

Abstract: Bilingual lexicons map words in one language to their translations in another, and are typically induced by learning linear projections to align monolingual word embedding spaces. In this paper, we show it is possible to produce much higher quality lexicons with methods that combine (1) unsupervised bitext mining and (2) unsupervised word alignment. Directly applying a pipeline that uses recent algorithms for both subproblems significantly improves induced lexicon quality and further gains are possible by learning to filter the resulting lexical entries, with both unsupervised and semi-supervised schemes. Our final model outperforms the state of the art on the BUCC 2020 shared task by 14 F1 points averaged over 12 language pairs, while also providing a more interpretable approach that allows for rich reasoning of word meaning in context. Further analysis of our output and the standard reference lexicons suggests they are of comparable quality, and new benchmarks may be needed to measure further progress on this task.

## introduction bilingual lexicons map words in one language to their translations in another, and can be automatically induced by learning linear projections to align monolingual word embedding spaces @xcite @xcite . although very successful in practice, the linear nature of these methods encodes unrealistic simplifying assumptions (e.g. all translations of a word have similar embeddings). in this paper, we show it is possible to produce much higher quality lexicons without these restrictions by introducing new methods that combine (1) unsupervised bitext mining and (2) unsupervised word alignment. we show that simply pipelining recent algorithms for unsupervised bitext mining @xcite and unsupervised word alignment @xcite significantly improves bilingual lexicon induction (bli) quality, and that further gains are possible by learning to filter the resulting lexical entries. improving on a recent method for doing bli via unsupervised machine translation @xcite , we show that unsupervised mining produces better bitext for lexicon induction than translation, especially for less frequent words. these core contributions are established by systematic experiments in the class of bitext construction and alignment methods (figure 1 ). our full induction algorithm filters the lexicon found via the initial unsupervised pipeline. the filtering can be either fully unsupervised or weakly-supervised: for the former, we filter using simple heuristics and global statistics; for the latter, we train a multi-layer perceptron (mlp) to predict the probability of a word pair being in the lexicon, where the features are global statistics of word alignments. in addition to bli, our method can also be directly adapted to improve word alignment and reach competitive or better alignment accuracy than the state of the art on all investigated language pairs. we find that improved alignment in sentence representations @xcite leads to better contextual word alignments using local similarity @xcite . our final bli approach outperforms the previous state of the art on the bucc 2020 shared task @xcite ) by 14 f 1 points averaged over 12 language pairs. manual analysis shows that most of our false positives are due to the incompleteness of the reference and that our lexicon is comparable to the reference lexicon and the output of a supervised system. because both of our key building blocks make use of the pretrainined contextual representations from mbart @xcite extraction cooccurrence(good, guten) = 2 one-to-one align(good, guten) = 2 many-to-one align(good, guten) = 0 cosine_similarity(good, guten) = 0.8 inner_product(good, guten) = 1.8 count(good) = 2 count(guten) = 2 lexicon induction multi-layer perceptron 𝑃 good, @xmath0.95 2020) and criss @xcite , we can also interpret these results as clear evidence that lexicon induction benefits from contextualized reasoning at the token level, in strong contrast to nearly all existing methods that learn linear projections on word types. ## related work bilingual lexicon induction (bli). the task of bli aims to induce a bilingual lexicon (i.e., word translation) from comparable monolingual corpora (e.g., wikipedia in different languages). following @xcite , most methods train a linear projection to align two monolingual embedding spaces. for supervised bli, a seed lexicon is used to learn the projection matrix @xcite @xcite . for unsupervised bli, the projection matrix is typically found by an iterative procedure such as adversarial learning @xcite , or iterative refinement initialized by a statistical heuristics @xcite . @xcite show strong gains over previous works by word aligning bitext generated with unsupervised machine translation. we show that retrieval-based bitext mining and contextual word alignment achieves even better performance. word alignment. word alignment is a fundamental problem in statistical machine translation, of which the goal is to align words that are translations of each in within parallel sentences @xcite . most methods assume parallel sentences for training data @xcite @xcite . in contrast, @xcite propose simalign, which does not train on parallel sentences but instead aligns words that have the most similar pre-trained multilingual representations @xcite . simalign achieves competitive or superior performance than conventional alignment methods despite not using parallel sentences, and provides one of the baseline components for our work. we also present a simple yet effective method to improve performance over simalign (section 5). bitext mining/parallel corpus mining. bitext mining has been a long studied task @xcite @xcite . most methods train neural multilingual encoders on bitext, which are then used with efficent nearest neighbor search to expand the training set @xcite @xcite . recent work has also shown that unsupervised mining is possible @xcite . we use criss @xcite foot_0 as one of our component models. ## baseline components we build on unsupervised methods for word alignment and bitext construction, as reviewed below. ## extension to word alignment the idea of using an mlp to induce lexicon with weak supervision (section 4.2) can be directly extended to word alignment. let @xmath1, t i } n @xmath2, and such features can help exclude such pairs. 7 not to be confused with joint probability. algorithm 1: inference algorithm for weakly-supervised lexicon induction. input: thresholds δ, n, model parameters θ, source words s output: induced lexicon l l ← ∅ for @xmath3, t 1 , . . . , s, t k ) ← bilingual word pairs sorted by the descending order of for a pair of bitext, simalign with a specified inference algorithm produces word alignment @xmath4, b i } i , denoting that the word tokens s a i and t b i are aligned. sabet et al. (2020) has proposed different algorithms to induce alignment from the same similarity matrix, and the best method varies across language pairs. in this work, we consider the relatively conservative (i.e., having higher precision) argmax and the higher recall itermax algorithm @xcite , and denote the alignments by a argmax and a itermax respectively. we substitute the non-contextualized word similarity feature (section 4.2) with contextualized word similarity where the corresponding word embedding is computed by averaging the final-layer contextualized subword embeddings of criss. the cosine similarities and dot-products of these embeddings are included as features. instead of the binary classification in section 4.2, we do ternary classification for word alignments. for a pair of word tokens s i , t j , the gold label y s i ,t j is defined as intuitively, the labels 0 and 2 represents confident alignment or non-alignment by both methods, while the label 1 models the potential alignment. the mlp takes the features x s i ,t @xmath5, and compute the probability of each label y by on the training stage, we maximize the log-likelihood of groundtruth labels: on the inference stage, we keep all word token pairs s i , t j that have ## experimental setup and baselines throughout our experiments, we use a two-layer perceptron with the hidden size of 8 for both lexicon induction and word alignment. we optimize all of our models using adam (kingma and ba, 2015) with the initial learning rate 5 × 10 -4 . for our bitext construction methods, we retrieve the best matching sentence or translate the sentences in the source language wikipedia; for baseline models, we use their default settings. for evaluation, we use the bucc 2020 bli shared task dataset @xcite and metric (f 1 ). like most recent work, this evaluation is based on muse @xcite . 8 we primarily report the bucc evaluation because it considers recall in addition to precision. however, because most recent work only evaluates on precision, we include those evaluations in appendix d. we compare the following baselines: bucc. best results from the bucc 2020 @xcite for each language pairs, we take the maximum f 1 score between the best closed-track results @xcite and open-track ones @xcite . our method would be considered open track since the pretrained models used a much larger data set (common crawl 25) than the bucc 2020 closedtrack (wikipedia or wacky; @xcite . vecmap. popular and robust method for aligning monolingual word embeddings via a linear projection and extracting lexicons. here, we use the standard implementation foot_5 with fasttext vectors @xcite foot_6 trained on the union of wikipedia and common crawl corpus for each language. 11 we include both supervised and unsupervised versions. wm. wikimatrix @xcite foot_8 is a dataset of mined bitext. the mining method laser @xcite ) is trained on real bitext and then used to mine more bitext from the wikipedia corpora to get the wikimatrix dataset. we test our lexicon induction method with wikimatrix bitext as the input and compare to our methods that do not use bitext supervision. ## word alignment results we evaluate different word alignment methods ( ## discussion we present a direct and effective framework for bli with unsupervised bitext mining and word alignment, which sets a new state of the art on the task. from the perspective of pretrained multilingual models @xcite @xcite , our work shows that they have successfully captured information about word translation that can be extracted using similarity based alignment and refinement. although bli is only about word types, it strongly benefits from contextualized reasoning at the token level.