Classification-Based Self-Learning for Weakly Supervised Bilingual Lexicon Induction (2020)

Abstract: Effective projection-based cross-lingual word embedding (CLWE) induction critically relies on the iterative self-learning procedure. It gradually expands the initial small seed dictionary to learn improved cross-lingual mappings. In this work, we present ClassyMap, a classification-based approach to self-learning, yielding a more robust and a more effective induction of projection-based CLWEs. Unlike prior self-learning methods, our approach allows for integration of diverse features into the iterative process. We show the benefits of ClassyMap for bilingual lexicon induction: we report consistent improvements in a weakly supervised setup (500 seed translation pairs) on a benchmark with 28 language pairs.

## introduction and motivation cross-lingual word embeddings (clwes), that is, representations of words in a shared crosslingual vector space, enable multilingual modeling of meaning and facilitate cross-lingual transfer for downstream nlp tasks @xcite . one of their primary use cases is bilingual lexicon induction (bli), that is, learning translation correspondences across languages which benefit the development of core language technology also for resource-poor languages and domains @xcite @xcite @xcite . earlier work focused on joint clwe induction from bilingual corpora, relying on word- @xcite , sentence- @xcite @xcite , or document-level supervision @xcite . however, recent focus is predominantly on post-hoc alignment of independently trained monolingual word embeddings: the * equal contribution. we extensively evaluate our classification-based self-learning procedure, termed classymap, on the standard bli data set @xcite spanning 28 pairs of diverse languages. the integration of the proposed self-learning method into vecmap @xcite , a state-of-the-art projection-based clwe framework, yields substantial gains over previous self-learning procedures. 2 we demonstrate that the improvements are indeed achieved through the synergy of diverse features used by the classifier. we also demonstrate further bli improvements when we treat bli as a supervised classification-based task. our departure point is a standard self-learning setup from related work @xcite , outlined in the following. at each iteration k, the dictionary d (k) is first used to learn the joint space algorithm 1: classification-based self-learning x1, x2 ← monolingual embeddings of l1 and l2 d ← initial word translation dictionary c ← trainclassifier(d) w1, w2 ← alignembeddings(x1, x2, d) for each of n iterations do d1,2 ← nn(w1, w2); d2,1 ← nn(w2, w1) d ← (d1,2 ∩ d2,1) \ d sort d descending by frequency d ← first p elements of d generate scores for each pair in d using c sort d descending by score add first k elements of d to d c ← trainclassifier(d) w1, w2 ← alignembeddings(x1, x2, d) return: w1 (and/or w2) and c the nearest neighbours in w (k) cl are then used to extract the new dictionary d (k+1) . previous work typically relies on a variant of mutual nearest neighbours in the aligned embedding space of the current iteration to select likely translation candidates for the next. however, as hinted by @xcite , that procedure still results in many noisy candidates inserted in the extended seed sets, and the error may get amplified over subsequent iterations. new self-learning procedure. therefore, we propose a more versatile self-learning process. we train a supervised classifier in each iteration: given a word pair, it produces a probability score denoting to which extent the pair is a correct translation pair. the classifier can be fed a wide range of features on the character, subword, and word level. we apply the classifier in two ways. first, at iteration k the classification scores are used to select likely translation candidates which are added to the dictionary d (k+1) for iteration k + 1. second, similar to @xcite , at test time we use the classifier scores to rerank translation candidates produced by 1) finding nearest neighbours in the final aligned embedding space and 2) considering orthographically similar candidates. 3 a high-level overview of the proposed classification-based selflearning procedure is outlined in algorithm 1. self-learning: components. for implementing the alignembeddings operation (see algorithm 1) we rely on the vecmap foot_1 system @xcite in its supervised variant. the nn function returns word pairs that are nearest neighbours in a given aligned embedding space. the trainclassifier functionality can be instantiated using any standard classification framework. in this work, we opt for a simple a multi-layer perceptron with a single hidden layer. a very important design choice concerns generating negative training examples for the classifier. all word pairs in the dictionary at current iteration d (k) are used as positive examples. for each positive pair (s, t), we generate two negative examples: 1) (s, x), where x is sampled uniformly from n o target words which are orthographically (measured by edit distance) most similar to s; 2) (s, y), where y is sampled uniformly from n c target words closest (by cosine) to s in the current space w (k) cl . this strategy performed considerably better than randomly generating negative examples. the intuition is as follows: at test time the classifier must operate on word pairs that are generated using nearest neighbour search. such word pairs are not random, but are rather very close in the aligned embedding space and are often orthographically similar. thus, this strategy for generating negative samples makes the train conditions for the classifier better reflect the test conditions. features. the classification-based approach allows for the integration of a wide spectrum of diverse features that capture different word translation evidence. we outline the sets of features used in this work, computed for each word pair (s, t). f1. edit distance -levenshtein and jaro-winkler distance between s and t @xcite . following @xcite we also include normalized edit distance, log of the rank of t in a list sorted by edit distance with respect to s, as well as a product of these two values. f2. cosine similarity of s and t in w (k) cl (at iter k). f3. aligned embeddings of s and t, pca-reduced to 10 dimensions (20 features in total). f4. normalized n-gram overlap @xcite ; f5. character n-grams -we extract all character ngrams and use χ 2 feature selection to select the 10 most indicative ones. the intuition is to allow the model to recognize indicative prefixes or suffixes. f6. subword-level similarity -we use multilingual subword embeddings (swes) based on bpes @xcite . we add the following features: i) we average the bpes of s and t and calculate cosine similarity of the resulting vectors, ii) the pairwise maximum cosine similarity of all pairs of swes (one from s and the other from t), and iii) the earth mover's distance between the two sets of swes @xcite . f7. frequencies -we provide the rank of the word in a list of all words sorted by frequency. the ranks are normalized by the number of words. at test time, if we use the classifier to perform the final reranking, we take for each source word s a set of candidate target word translations as the union of 1) the top n ro target word neighbours of s by edit distance, and 2) the top n rc target word neighbours of s by cosine in the final aligned w cl . we then score the n ro + n rc candidates using the classifier from the last self-learning iteration. ## conclusion and future work we introduced classymap, a novel classificationbased approach to self-learning, which is a crucial component of projection-based cross-lingual word embedding induction models in low-data regimes. we reported its usefulness and robustness across a wide spectrum of diverse language pairs in the bli task, confirming the usefulness of learning classifiers both as part of the self-learning procedure as well as for the final word retrieval in the bli task. this proof-of-concept work opens up a wide spectrum of interesting avenues for future research, including the use of more powerful classifiers, more sophisticated features (e.g., character-level transformers), and fine-grained linguistic analyses on the importance of disparate features over different language pairs. one particularly exciting direction is the application of our classification-based self-learning framework on top of the most recent methods that induce bilingual spaces via non-linear alignments @xcite . the code is available online at: