Mining Bilingual Word Pairs from Comparable Corpus using A pache Spark Framework (2021)

Abstract: Bilingual dictionaries are essential resources in many areas of natural language processing tasks, but resource-scarce and less popular language pairs rarely have such. Efficient automatic methods for inducting bilingual dictionaries are needed as manual resources and efforts are scarce for low-resourced languages. In this paper, we induce word translations using bilingual embedding. We use the Apache Spark framework for parallel computation. Further, to validate the quality of the generated bilingual dictionary, we use it in a phrase-table aided Neural Machine Translation (NMT) system. The system can perform moderately well with a manual bilingual dictionary; we change this into our inducted dictionary. The corresponding translated outputs are compared using the Bilingual Evaluation Understudy (BLEU) and Rank-based Intuitive Bilingual Evaluation Score (RIBES) metrics.

## introduction digitised bilingual dictionaries primarily exist for resource-rich language pairs, such as english-german, english-chinese, english-hindi, etc. @xcite . such dictionaries are helpful for many natural language processing (nlp) tasks such as machine translation (mt) for translating out-of-vocabulary (oov) words, cross-lingual information retrieval, cross-lingual word embedding and multilingual parts-of-speech tagging @xcite @xcite . creating a bilingual dictionary requires high-quality parallel corpora and expert linguists, both of which are scarce and costly in resource-poor languages @xcite . previous works focus on methods that were based on pivot languages @xcite @xcite , aligning words @xcite ; tufis ¸and maria @xcite or using dependency relations @xcite . the pivot-based dictionary induction is a contemporary method that uses only dictionaries to and from a pivot language (intermediate language) to generate a new dictionary. this method is not very effective for highly ambiguous languages as it yields highly noisy dictionaries because lexicons of a language do not exhibit transitive relationship @xcite . word alignment systems identify the translation equivalence of lexical units between two sentences that are sentence aligned @xcite . depending on the purpose, the system may focus on the specific lexical units, e.g. a single word or collocation @xcite @xcite . the dependency relation method is based on the premise that related words in different languages have a similar dependency relationship. these methods require either excellent linguistic knowledge or linguistic resource. the research line has robust outcomes on bilingual lexicon induction with the evolution of word embedding either by independently aligning trained word embedding in two languages or using the bilingual embedding to induce word translation pairs through nearestneighbour or similar retrieval methods. in the bdi task, given a list of 'n' source language words w s 1 , w s 2 , ...w sn , the goal is to determine the most appropriate translationw t i , for each query word w s i . finding a target language word embedding wv t i is accomplished by computing the nearest neighbour to the source word embedding wv s i in the shared semantic space, where cosine similarity is a measure between the embedding @xcite . however, this creates a phenomenon called hubness. in high-dimensional spaces, some data points, called hubs, are extraordinarily close to many other data points @xcite ; this results in inappropriate/noisy translation. in this paper, a simple cartesian product of the bilingual/cross-lingual word embedding is used and filters the product outcome based on some linguistic regularities and thresholds. the generated (inducted) bilingual dictionary is used as a separate phrase-table in an nmt system. the system produces translations for every word in the text; the translations are validated for quality using the bilingual evaluation understudy (bleu) and rank-based intuitive bilingual evaluation score (ribes) metric. ## bilingual and cross-lingual word embedding in this paper, the terms 'bilingual' and 'crosslingual' for word embedding is used with varying notions. the bilingual embedding maps the source and target language embedding in the shared semantic space. in contrast, the cross-lingual embedding learns a transfer function to translate the embedding from the source language semantic space to target language space; this preserves the more actual semantics pertained to that language @xcite . visualisation of the embeddings is shown in figure 1 and figure 2 . bilbowa toolkit @xcite is used to generate bilingual word embedding. the embedding of source and target language are trained jointly so that related words of two languages are closer to each other in the shared space. therefore, the translational equivalence has higher cosine similarity. the model is trained with minimal parallel corpus and large monolingual corpora. however, the cross-lingual embedding is learned with a very bare minimal resource as small as 5000 source-target word pairs. global neighbourhood is estimated as cross-lingual entropy. the main advantage of this method over bilingual embedding is that it is possible to generate embedding in the target language semantic space instead of shared space. in shared semantic space, the most semantic information pertained to the language is lost and likely to infer word vectors for related languages. ## implementation the embedding size of the english word list @xmath0. tamil has more number words compared to english because of the inflected forms. the dimension of the cartesian product of the word pair list (english and tamil) is 90812418 × 300; this takes months for a typical computer system to compute. this complex computation is deployed to the cluster using apache spark ® framework @xcite . the word pairs are filtered in two folds, cosine similarity and lemmatization @xcite , where the root word is extracted from the surface forms. in the case of cross-lingual embedding, cross-lingual entropy is used instead of the cosine similarity measure. figure 3 shows the architecture. the word embedding of source and target language is mapped to a key-value pair resilient distributed datasets (rdds), a fundamental data structure of spark; the word being a key and 300dimensional representation as values. the cartesian product of two rdds (en rdd and ta rdd) generates the pair rdd. on the pair rdd, cosine similarity or cross-lingual entropy is applied to filter top similar words. filtered rdd is further refined using a lemmatizer to avoid the inflected terms. the resultant rdd is saved as text file; this has the most similar source and target word, a bilingual dictionary. the opennmt framework @xcite is used for training an nmt system with the training parameter as shown in ## corpora description for the training language model, the monolingual tamil corpus from the centam dataset (p. et al., 2020) is used. likewise, for training the machine translation systems, the english-tamil parallel cor-pus from the centam dataset is utilised. the specifics of the centam corpus used is reported in ## conclusion and summary in this paper, we generated an english-tamil bilingual dictionary using both bilingual (vectors in the same space) and cross-lingual (vectors in separate space, mapped) word embedding. in order to validate this induced dictionary, we have employed a table driven neural machine translation (nmt) system. the goal was to measure the quality of the translated output (tamil as the target language) when the original manual dictionary (mandic) is replaced with the induced dictionary (indic). the baseline nmt system was trained on english-tamil parallel corpus with over 56000 entries. a testset with 700 aligned sentences was used for validation. the translation quality is measured over the reference translations which are available (aligned tamil sentences). eventually, we will have three categories of translated output, namely, baseline, mandic and indic. we compare each of them with the reference translation using the ribes and bleu metric @xcite to ascertain their quality. it is important to note that the quality of the translations is not of our interest but the change in performance when using different dictionaries. ribes is used as the scoring model as it is invariant to word order and morphology @xcite . our results suggest that the induced dictionary performs at par or better than the original manual dictionary. this is also due to the fact that the lexicons are rendered in a context-sensitive manner from word embedding. the lookup process is implemented using apache spark ® framework in scala language. induction is a simple reverse lookup using the cartesian product of all bilingual embedding. the size of this cartesian product matrix is 1 × 10 7 × 300 values which makes it highly computational. apache spark can run in parallel, hence, accelerate time and optimise memory. in this paper, bilingual embedding generated by bil-bowa @xcite is mainly used, but this methodology is also tested with cross-lingual embedding and found equally effective @xcite ). the differences between them are: bilingual embeddings are generated from parallel and good quality comparable bilingual corpus, whereas cross-lingual embedding can be learned from minimal bilingual data. learning such cross-lingual embedding for resource-poor languages can help to generate induced dictionary resources of even unknown words with a fair amount of accuracy.