Bilingual Lexicon Induction across Orthographically-distinct Under-Resourced D ravidian Languages (2020)

Abstract: Bilingual lexicons are a vital tool for under-resourced languages and recent state-of-the-art approaches to this leverage pretrained monolingual word embeddings using supervised or semi-supervised approaches. However, these approaches require cross-lingual information such as seed dictionaries to train the model and find a linear transformation between the word embedding spaces. Especially in the case of low-resourced languages, seed dictionaries are not readily available, and as such, these methods produce extremely weak results on these languages. In this work, we focus on the Dravidian languages, namely Tamil, Telugu, Kannada, and Malayalam, which are even more challenging as they are written in unique scripts. To take advantage of orthographic information and cognates in these languages, we bring the related languages into a single script. Previous approaches have used linguistically sub-optimal measures such as the Levenshtein edit distance to detect cognates, whereby we demonstrate that the longest common sub-sequence is linguistically more sound and improves the performance of bilingual lexicon induction. We show that our approach can increase the accuracy of bilingual lexicon induction methods on these languages many times, making bilingual lexicon induction approaches feasible for such under-resourced languages.

## introduction bilingual lexicon induction (bli) is the process of creating lexicons for two or more languages from monolingual corpora @xcite . it is a time-consuming process to do it manually so automatically inducing bilingual lexicons based on edit-distance @xcite , comparable corpora @xcite , bilingual corpora @xcite or pretrained embeddings from monolingual corpora @xcite is more suitable. however, sentence-aligned parallel data is not available for all languages. methods based on unsupervised or semi-supervised learning can utilise readily available monolingual data to induce bilingual lexicons. @xcite showed that an iterative self-learning method could bootstrap this approach without the need of a seed dictionary by utilising numbers as seed dictionary through adversarial training. however, @xcite showed that with even a small seed dictionary, the results could be improved considerably. this task is further complicated by the fact that many languages use distinct scripts, and as such learning the similarities between cognates is a non-trivial task. as such, bli is a challenging task for under-resourced languages due to lack of seed dictionaries and large monolingual corpora. for this work, we proposed to use the indowordnet as a seed dictionary for the closely related dravidian languages, namely tamil, telugu, kannada, and malayalam, which use different scripts. bli between closely-related languages has shown to perform better than unrelated languages @xcite , since closely related languages often share similar linguistics properties and cognates @xcite . cognates are words that have a similar meaning and similar orthography based on etymological relationships @xcite . computational models of monolingual embeddings also exhibit isomorphism across closely related languages @xcite based on the assumption that word embeddings in different languages have approximately the same structure. this isomorphic property was exploited by @xcite and @xcite to map monolingual word embeddings in different languages to a shared space through a linear transformation. for closely-related languages, it follows that cognates can be used as a form of alignment as words that have a similar form are quite likely to be cognates and therefore could be used as a weak seed dictionary. given previous work on the use of seed dictionaries @xcite , the usage of such alignments is likely to improve performance of bli. previous works used the levenshtein distance @xcite ; however, this is not linguistically well-motivated as it allows for multiple changes that are not consistent with the kinds of changes seen etymologically. the goal of this work is to exploit the orthographic information between languages that use a different script. for that purpose, we bring the languages into a single script, which allows us to take advantage of the cognate properties of closely related languages. this paper has two principal contributions: first, we study the use of transliteration, and we demonstrate that it is an effective and necessary step which yields more isomorphic embeddings and obtains more robust bli. second, we show that the use of the longest common subsequence (lcs) is a superior method of assessing the cognate similarity. ## dravidian languages dravidian languages are the common terminology @xcite used to represent the south indian languages, which consist of around 26 languages divided into four branches: 11 in the southern group, 7 in the south-central group, 5 in the central group and 3 in the northern group @xcite . out of the 26 dravidian languages, many of them are non-literary languages except the four languages chosen for this paper. indigenous minority populations primarily use the non-literary languages. the modern society widely uses the four literary languages in literature, public communications, government institutions, academic settings and many other places in the day-to-day life of an ordinary person @xcite . for many natural language processing tasks such as machine translation (mt) systems, it is essential to have a corpus of written documents, as well as well-defined lexicons and grammar for the selected languages @xcite . hence in this work, we will focus on the four chosen dravidian languages tamil, malayalam, kannada and telugu which are spoken by approximately 210 million people (steever, 2015) across the world either as their first or second language. even though these languages share a common root, they cannot be termed as regional dialects of a language of the same origin @xcite . tamil and malayalam are more closely related such that a regional speaker of one language can understand another without translation @xcite . figure 1 illustrates the words for 'salt', 'coin', 'vehicle', 'walk' are similar in both tamil and malayalam. the tamil, malayalam and telugu languages have their own written script symbols whereas telugu and kannada have significant similarities in their script symbols. though telugu and kannada have these similarities, they are not readily intelligible for speakers of the language. the study of languages suggests that these languages formed a single language around late 4000 bce and then started evolving on their own @xcite . since the languages evolved sharing geographical, etymological and political borders, the cognates may have evolved similar meanings or borrowed words from each other. chakravarthi et al. (2019a) have compared the latin script and the international phonetic alphabet (ipa) for multilingual translation systems and shown that bringing the dravidian languages into latin script outperforms a multilingual neural machine translation system trained on native script and ipa. inspired by this, we transform the dravidian language monolingual corpora into a single script (latin script). ## experimental settings words that share a similar context are semantically related. based on their word embeddings, methods represent words in a vector space by grouping semantically similar words near each other. word embeddings are useful for several lexical-semantic tasks such as detecting synonyms and disambiguating word sense. several pre-trained embedding models are publicly available such as word2vec foot_3 @xcite , global word representation-based models (glove) foot_4 @xcite and fasttext foot_5 @xcite . fasttext was used to create monolingual embeddings from wikipedia articles. fasttext enhances traditional word-based vectors by representing each word as a bag of character n-grams. incorporating this subword information from fasttext embeddings as well as semantic relatedness allows the capturing of orthographic and morphological similarity. we did not use the pre-trained embeddings from fasttext since we also created embedding for transliteration. given that the main focus of our work is on bringing closely related languages into a single script, we transcribed the wikidump corpus before creating word embeddings and training seed dictionary. we conducted an experiment on bli on language pairs, tamil-telugu, tamil-kannada, tamil-malayalam, telugu-malayalam, telugu-kannada, and kannada-malayalam. similar to riley and gildea (2018) foot_6 , we stop training when the improvement on the average cosine similarity for the induced dictionary is below 10 -6 between successive iterations. we compare our methods with baselines with and without a seed dictionary. an automatically-generated dictionary consisting only of numeral identity translations such as 4-4, 8-8, as in @xcite , was used as a training set as the input dictionary to the baseline system without a seed dictionary. ## results and discussion we show results of all eight cases studied for bli in once the dictionary d is learned by the self-learning process, we use the dictionary d to create translations for source words from the test set, and compare it against the target words within the evaluation set to calculate translation accuracy. translation accuracy is the proportion of correct predictions among the total number of cases examined in the test set given in equation 3 . the total number of cases examined for language pairs under study is given in ## conclusion in this paper, we have explored bringing closely related languages into a single script and their impact on the task of bli from monolingual word embeddings. we created a seed dictionary for tamil, telugu, malayalam and kannada from indowordnet. our initial experiments with 'off-the-shelf' bli, based on the alignment of numbers, produced very poor results, showing that these methods were not possible to apply directly to these under-resourced languages. we found that mapping these languages to a single character system helps the system to discover cognates in these closely-related under-resourced languages. further, we showed that lcs is more linguistically sound for cognate detection by quantitative evaluation and in application to bli. this paper shows that importance of evaluating methodologies directly on under-resourced languages as the challenges related to these languages may require modification to existing methodologies so as to make them work effectively, as demonstrated in this paper.