Combining Word Embeddings with Bilingual Orthography Embeddings for Bilingual Dictionary Induction (2020)

Abstract: Bilingual dictionary induction (BDI) is the task of accurately translating words to the target language. It is of great importance in many low-resource scenarios where cross-lingual training data is not available. To perform BDI, bilingual word embeddings (BWEs) are often used due to their low bilingual training signal requirements. They achieve high performance, but problematic cases still remain, such as the translation of rare words or named entities, which often need to be transliterated. In this paper, we enrich BWE-based BDI with transliteration information by using Bilingual Orthography Embeddings (BOEs). BOEs represent source and target language transliteration word pairs with similar vectors. A key problem in our BDI setup is to decide which information source – BWEs (or semantics) vs. BOEs (or orthography) – is more reliable for a particular word pair. We propose a novel classification-based BDI system that uses BWEs, BOEs and a number of other features to make this decision. We test our system on English-Russian BDI and show improved performance. In addition, we show the effectiveness of our BOEs by successfully using them for transliteration mining based on cosine similarity.

## introduction the task of bilingual dictionary induction is defined as finding target language translations of source language words. it is an important building block in the area of machine translation (mt) and it is one of the main tasks for bilingual word embedding evaluation @xcite . recent work shows that good performance can be achieved relying only on bwes, which can be built with only a weak bilingual signal, such as a small seed lexicon of a few thousand word pairs @xcite or common tokens in the source and target languages @xcite . in addition, they can even be built without any bilingual signal @xcite , making them the basis of unsupervised mt systems @xcite . standard bdi learns word representations based on approaches that exploit solely word-level information such as word2vec @xcite or fasttext @xcite and then map them to a shared bwe space. although bwe-based approaches show high bdi performance, they struggle with a subset of hard-to-translate words such as named entities for which orthographic information should be used instead of semantic information. several approaches have integrated orthographic information into the bdi system. @xcite relied on character-level information in their classification based bdi system by using an rnn architecture. @xcite combined orthographic information with bwe-based word similarity information using an ensembling method. both of these approaches showed improved results, but they relied on levenshtein distance to get translation candidates for a source word during prediction, which is not applicable for language pairs with different scripts. to bridge the gap between languages with different scripts, a transliteration system was employed by @xcite . they followed the approach of @xcite but used the transliteration system instead of levenshtein distance to get candidates used in the ensembling model. on the other hand, as they also showed, the ensembling approach often fails to decide correctly if a word has to be transliterated or not; this is because there are only two independent scores available, the score of the (semantic) bwe, and the score of the (orthographic) transliteration model. in this paper, we present our novel approach to bdi focusing on words that have to be transliterated to another script, which is especially important for low-frequency words, but also relevant for highfrequency named entities. our aim is to improve bdi systems in two respects: (i) eliminating the need for language specific orthographic information, such as is used in @xcite to be able to better decide when to choose transliteration over semantic translation. we propose a new approach for language pairs with different scripts by combining semantic information with orthographic information. for the latter we introduce bilingual orthographic embeddings (boes) of words, which represent transliteration pairs in the source and target language with similar vectors. we build boes using a novel transliteration system trained jointly for both language directions. we refer to this novel system as seq2seqtr. seq2seqtr can also be used to extract candidate transliterations for a source word. it is applicable to any language pair (as opposed to levenshtein distance). to make a more informed decision about which words should be transliterated (which means we shoud primarily trust the boes) and which should be semantically translated (which means we should primarily trust the bwes), we use a classification approach similar to @xcite , exploiting our pretrained encoder from seq2seqtr. in contrast to their approach, we use additional features, such as frequency, length, similarity scores, and the ranks assigned by the semantic and character-level submodels, and show that they are necessary to make the right decision. we test our system on the english-russian (en-ru) data provided in the bucc 2020 shared task @xcite . test dictionaries were released in three frequency categories: high, middle and low. we evaluate our system on all three sets, both separately and jointly, and show improved performance on all three frequency ranges compared with previous approaches. furthermore, we show that our classification system is more robust than the ensembling of @xcite , which required specialized tuning on each frequency set. lastly, we conduct a further analysis of the quality of the proposed boes by running transliteration mining on the news 2010 shared task data @xcite by using the vector similarity of bilingual orthographic embeddings of words. we show good performance on the task indicating the usefulness of boes for other downstream tasks. ## approach to tackle the bdi task we exploit bwes, character-level information (in the form of boes) and manually engineered features -such as word frequency and length -and integrate them into a classifier that predicts if the word pair is a translation or not. we extract candidate translations for a source word based on (i) bwes and (ii) seq2seqtr (our transliteration model) as described in the following sections. finally, we rerank the two groups of candidates with our classification system described below and take the top ranked candidates as our prediction. ## experimental setup we ran our bdi experiments on the bucc 2020 shared task dataset @xcite ; it provides both monolingual corpora and bilingual dictionaries for english-russian. since the official test set of the shared task is undisclosed, we relied on the released training set, a random subset of the muse dictionaries foot_2 released by @xcite . it is divided into three subsets, high, middle and low frequency, containing words ranked between 1-5000, 5001-20000 and 20001-40000 in the original dictionary, each having 2000 unique foot_3 source words. we split them into train, development and test sets (70%/15%/15%). we run experiments on the three frequency sets separately and jointly. we followed the official setup of the bucc shared task and relied on the wacky corpora @xcite as monolingual data to get the full language vocabularies and their frequencies. as mwes we used the fasttext skipgram models @xcite released by the shared task organizers, which were trained using the wacky corpora with the following parameters: minimum word count 30; vector dimension 300; context window size 7; number of negatives sampled 10 and number of epochs 10. to align them we used vecmap @xcite in a supervised setup using only high frequency word pairs for training, since less frequent words can be detrimental for mapping quality @xcite . we use the bwes of the 200k most frequent words following @xcite . in addition, in an ablation study described below we used the 1000 word pairs of the news 2010 transliteration mining test set @xcite to test the quality of the boes. for seq2seqtr we use learning rate 0.01, batch size 32, hidden size 128, single encoder and single decoder layers with the "dot" attention method of @xcite that compares the states with their dot product score. as mentioned above we use the same architecture for building boes, but we use different parameters. we tuned the hidden size (2000) of the gru layers (used as the boes) and the character embedding size (300) along with the classifier on the bucc high, middle and low frequency dev sets jointly. for training the bdi classifier we used adam optimizer (kingma and ba, 2014) with learning rate 0.001, batch size 32, 2 hidden layers on top of the described features with hidden size 300, dropout 0.01 ## experiments and results in this section, we show the main results of our approach and compare them with the baselines. we also show an evaluation of the boes on the news 2010 shared task @xcite to better understand their quality on a dataset that only contains transliteration pairs. ## conclusion bilingual dictionary induction is a relevant task for many applications and it is an important building block in the area of mt. in this paper we described our system for bdi for language pairs with different scripts focusing on words for which semantic information alone is insufficient. we combined semantic and orthographic information via transliteration. our proposed model has the novel ability to make a reasonable decision on which source of information to choose via a classification approach that exploits -together with manually designed features -word embeddings and character-level information (boes). our novel boes were learned by a language agnostic transliteration system. we tested our system on the english-russian bucc 2020 dataset and we showed improved results compared to the baselines. we also showed that our model is able to re-rank the candidate words better in contrast to other approaches. we evaluated our system on high, middle and low frequency sets separately and jointly. finally, we evaluated our boes by running transliteration mining on the news 2010 dataset, showing that they achieve good performance even if they were not meant for that specific task. also, boes were shown to be able to encode the orthographic structure of words independent of the language. that means that they are universal embeddings that represent transliteration pairs similarly -a property that is useful for other downstream tasks as well. all in all, we presented a system able to combine word-level information with character-level information by means of transliteration and classification models for bdi that improved the baseline results.