Using Graph-Based Methods to Augment Online Dictionaries of Endangered Languages (2022)

Abstract: Many endangered Uralic languages have multilingual machine readable dictionaries saved in an XML format. However, the dictionaries cover translations very inconsistently between language pairs, for instance, the Livonian dictionary has some translations to Finnish, Latvian and Estonian, and the Komi-Zyrian dictionary has some translations to Finnish, English and Russian. We utilize graph-based approaches to augment such dictionaries by predicting new translations to existing and new languages based on different dictionaries for endangered languages and Wiktionaries. Our study focuses on the lexical resources for Komi-Zyrian (kpv), Erzya (myv) and Livonian (liv). We evaluate our approach by human judges fluent in the three endangered languages in question. Based on the evaluation, the method predicted good or acceptable translations 77% of the time. Furthermore, we train a neural prediction model to predict the quality of the automatically predicted translations with an 81% accuracy. The resulting extensions to the dictionaries are made available on the online dictionary platform used by the speakers of these languages.

## introduction for many endangered languages there are several existing dictionaries and other bilingual lexical resources for different language pairs. for example, for many uralic languages there are german dictionaries, as that has traditionally had a strong role as a scientific language of the field. also the dictionaries in local majority languages such as finnish, estonian, latvian and russian are very common. although the fact that a great many of them exist only as printed copies limits their use in the digital era. nevertheless, dictionaries play an important role in language documentation and revitalization ef-forts. for endangered uralic languages, akusanat online dictionary @xcite collects multilingual dictionary resources in multiple endangered languages such as the ones in focus of our paper: komi-zyrian, livonian and erzya. making it possible for native speakers and language learners to access such a resource has a very big societal impact within the language communities. furthermore, online resources such as wiktionary have gathered very large amounts of lexical data for majority languages. this data does not necessarily represent a fully curated and finalized product in which all entries would be of an equal quality. only more recently has there been interest in building such resources in the languages that are nowadays more widely used, such as english. as creating these resources is an enormous undertaking, we investigate in this study the possibility of predicting translations from endangered languages to resource-rich languages automatically from existing translations in these high-resource language wiktionaries. we would like to point out that the languages we are working with in this paper are endangered, not just low-resourced (see @xcite . according to unesco atlas of world languages (moseley, 2010), komi-zyrian (kpv) has 217,316 native speakers and erzya (myv) 400,000 native speakers. livonian (liv), however, does not have any surviving native speakers foot_0 , but has a small community of second language speakers. apart from livonian, these languages have received some digital language documentation interest. erzya @xcite and komi-zyrian @xcite have small universal dependencies tree banks and morphological transducers @xcite . in the method we investigate in this study, the translations for a word in different languages are represented as graphs. this allows for an effective use of a large number of lexical resources that are not complete, but support one another. our main contributions in this work are: our approach makes it possible for lexicographers to bootstrap new languages into existing multilingual dictionaries. this saves time as instead of building a lexicon from the ground up, their task becomes more of a post-editing, where new translations need only to be verified rather than written from scratch. in the context of larger languages, post-editing has become mainstream in lexicographic work (see @xcite , however in the context of endangered languages post-editing has thus far received less lexicographic interest. ## related work there is a plethora of nlp work out there relating to endangered languages ranging from rule-based approaches @xcite @xcite to latest neural models @xcite @xcite . in this section, however, we focus more on work on extending dictionaries. there has been several attempts in the past in predicting new translations in bilingual and multilingual dictionaries. in this section, we describe the most relevant ones to our work. there has been related approaches to extending semantic knowledge bases @xcite @xcite ), but we leave their detailed description out of this section as the problem the approaches try to solve is fundamentally different in terms of the availability and magnitude of the data. @xcite have proposed a method for reversing bidirectional dictionaries (e.g., reversing hindi-english to english-hindi). their approach requires wordnet foot_1 @xcite for at least one of the languages, and uses the similarities between the words and their synonyms, hyponyms and hypernyms in wordnet to estimate the quality of the reverse translations. they have tested the method by reversing resource-poor and endangered language dictionaries (e.g. karbi, hindi and assamese) to have english as the source language instead of the destination language. it is worth noting that this approach is not capable of producing dictionaries or translations in new languages. @xcite proposed a method for creating new dictionaries for resource-poor languages. in their work, a dictionary of a low-resource language to a resource-rich language with a high-quality wordnet is needed. to translate a word from the source language to a new language (e.g. arabic), their method uses links between the english word-net and existing multiple intermediate wordnets of other languages such as finnish and japanese to highlight the relevant words in the wordnets. thereafter, each of these words are translated to the desired destination language using existing machine translation systems such as google translate. the higher the agreement between multiple machine translation systems, the higher the score given to the translation. a constraint-based approach for inducting new bilingual dictionaries for low-resource languages that are share the language family has been proposed by @xcite . in their approach, a graph is constructed from two bilingual dictionaries (i.e. a-b and b-c, where b is the intermediate language), and new potential translation links are examined by treating the problem as conjunctive normal form (cnf) and using wpmaxsat solver to identify the new translations. this work has been extended further in @xcite to generalize the method to work for a larger group of languages and identify the best constraint set according to the language pairs. a graph-based method for combining multiple wiktionaries and inferring new translations using graph-based probabilistic inference measured by random walks was proposed by @xcite . the goal of their work is to construct a huge dictionary covering the well-resourced languages (e.g., english, french, spanish, . . . etc) and suggest new dictionary translations; nonetheless, their work does not address endangered or resourcepoor languages. another graph-based method was embraced by @xcite . @xcite have trained a support vector machine (svm) model to predict whether a new translation is valid or not. given multiple bilingual dictionaries, a directed graph is constructed where nodes are unique words with their language and part-of-speech tag. depth-first search is applied to find cycles in the graph. translations found in cycles with a translation in the dictionary from the target word back to the source are considered to be positive examples, whereas translations found in paths but not cycles are treated as negative instances. additional features are passed to the model as well, such as the frequency of source word in a dictionary, number of available paths between the source and target words, and, in the case of sharing the language family, the average levenshtein distance between all the words in the path. this method was not investigated nor evaluated for endangered languages. ## data two types of resources are used in our approach, 1) xml dictionaries of endangered languages (such as komi-zyrian, livonian and erzya, with kpv, liv and myv as iso 639-3 codes respectively) and 2) wiktionaries foot_2 of resource-rich languages (such as english and french). while we could utilize the finnish wordnet @xcite as an additional resource in this task as done in some of the previous work, however, in practice it would introduce more noise due to the relatively low quality of the finnish wordnet foot_3 . ## inferring new translation candidates representing translations in a graph, where words are represented as nodes and translations between words as edges, is intuitive and has been successfully used for the task in the past, as described in the related work. in fact, some of the modern approaches to lexicography have also rejected the traditional tree structure of a dictionary in favor of a graph representation @xcite . similarly, we represent both types of dictionaries, xmls and wiktionaries, in a graph-based network using net-workx library @xcite . unlike some of the previous work such as @xcite , the graph is not directional, given that nearly all lexical translations work bidirectionally. let @xmath0, e) denote the graph, where v is all the vertices/nodes in the graph and e is all undirected edges/links between two nodes. we initialize the graph with all translations from the five wiktionaries in such a way that their entries become interconnected based on words and their translations. to predict new translations from the source language s to the target language t , we load the xml dictionary of the desired endangered language to the graph while omitting any existing translations to the target language. this is done to ensure that all translations to the target language are projected by the method. once the graph is constructed, we iterate over all nodes from the source language v @xmath1. for all the neighbouring nodes linked to the source language n, we examine whether they belong to the target language, i.e. @xmath2. when such a constraint is satisfied, a new translation between the source lexeme s and n is considered as a candidate translation and assessed using link predictions methods. all candidates scoring zero on any of the link predictions methods described below are pruned out. we employ four link prediction methods to discover new translations; these are 1) jaccard coefficient @xcite , 2) adamic-adar index (adamic and adar, 2003), 3) resource allocation index @xcite , and preferential attachment score @xcite . in short, jaccard coefficient computes a score based on the common neighbours between the source and target nodes with respect to the total number of their neighbours. the adamic-adar index is defined as: the resource allocation index is defined similarly but without taking the log of the denominator. lastly, the preferential attachment score measures the magnitude of the neighbours of each node, which is defined as an example of a sub-graph containing the livonian lexeme (japān) along with links to existing translations in the xml dictionary (which are japani in finnish, jaapan in estonian and japāna in latvian) is shown in figure 2 . all the remaining nodes in the graph and their black connections to the other nodes are from wiktionaries. by running the link prediction methods described above to infer translations from livonian to english, two new links are suggested and they point to the lexemes japan and nippon, shown in red dashed lines. the methods were able to recommend the link to the japan with high confidence as there is a strong support based on their neighbouring nodes (i.e. liv_japāna, fin_japani and est_jaapan), whereas the link to nippon had a low confidence as only one node supports it (i.e. est_jaapan). ## manual evaluation in our evaluation, we run the link prediction method for the following four language pairs, 1) erzya and english 2) livonian and english, 3) komi-zyrian and english and 4) komi-zyrian and french. which resulted in 17,042, 22,911, 9,611 and 7,765 translation suggestions for the four language pairs, respectively. to evaluate the method, we have reached to fluent speakers in the source and target languages and requested them to manually annotate 200 randomly selected predictions. none of these predictions existed before in the xml dictionaries between each language pair. for each translation, they were instructed to indicate whether it is 1) good, 2) acceptable, 3) incomplete or 4) bad. good translations are dictionary-ready entries and can be automatically populated as they are. acceptable instances are correct predictions but may contain ambiguity due to, for example, synonymy or polysemy. incomplete translations are close to the desired translation but require manual modifications, while bad translations are completely off predictions and should be removed. in total, we obtained 800 annotated predictions. ## automatic detection of good predictions to further aid lexicographers in creating dictionaries, especially for endangered languages, we build an artificial neural network model for detecting whether a predicted translation by the methods is a good one. an automated way of filtering out the bad translations cuts the time needed for going through the predictions manually. we have experimented with different neural architectures and techniques. for the scope of this work, we describe the outperforming model which is a multilayer feedforward neural network (for a summary of the architecture, see in our model, we utilize adam optimizer (kingma and ba, 2014) and a sigmoid layer combined with binary cross entropy as the loss function due to its suitability for the binary classification task. to obtain the classification from the model, a sigmoid function followed by rounding the result is applied post inference. for the problem we are tackling, there are no available training datasets, neither for endangered languages nor resource-rich languages. to overcome this, we exploit our manual annotations and split them into 80-20 splits for training and testing. to convert the annotations into binary classes, we treat all good, acceptable and incomplete translations as positive instances and bad ones as negative. after 1,000 epochs of training with a learning rate of 0.001, the model reached an accuracy of 81%. ## discussion when looking at the bad candidate translations, the reasons why they were predicted by our method can be divided roughly into two categories: polysemy and wrong translations in the original xml dictionaries. a polysemy of a word in one language can cause a wrong translation to appear in another language that does not exhibit the same polysemy. for example the komi-zyrian word гол had been translated into paint instead of the correct translation goal. this is due to polysemy in finnish as the finnish word maali means both goal and paint. had there been more translations in between languages for these words that do not have the finnish polysemy, the graph based model would have been less likely to predict this translation. we have attempted to test the method by focusing solely on wiktionary data, where we would omit all existing translations from a particular source language to another (e.g., finnish to french or english). nonetheless, many of the predicted translations were good but were missing from the wiktionary of the source language, making it infeasible to assess the effectiveness of the method. despite that, this is a strong indication that the proposed method with our model could be employed to enrich existing wiktionaries further. an idea we had for training a neural model for predicting whether the new predictions are good or bad was to generate synthetic training data automatically. in practice, collecting examples of good translations from wiktionaries is easy, but producing automatically examples of bad translations is more difficult. predicting random links between words would result in all of the link prediction models outputting such a low score that it would hardly be representative of the real case of bad translations that are mainly due to polysemy or wrong initial translations. we tried out producing a dataset of bad translations with the idea that if an english word, for instance can is translated into voida (be able to) and purkki (can as a container) in finnish and võima (be able to) and purk (can as a container) in estonian, then predicting voida as a translation of purk and purkki as a translation of võima would make our synthetic data have very representative examples of bad translations. however, in practice, we ran into a coverage issue in wiktionaries. for example, the english wiktionary did not have any entry that would have had at least two translations into finnish and estonian. this made our good idea in theory impossible in practice. while quality and coverage of the existing data pose challenges, our work has provided some insight for the lexicographers working with these resources about the limitations of the current state of the lexical resources. this has been well received as a form of a sanity check among the lexicographers in question given that the lexicographic resources have been built by different people depending on their funding situation. this means that a lot of the work done in the dictionaries has been there before the current people working with the resources have started extending them. in our graphs, we have omitted the part-ofspeech because it is not present for all lexemes, whether in the xml dictionaries or wiktionaries. taking them into consideration would have resulted in inferring low-quality translations in smaller magnitudes. therefore, we believe that incorporating part-of-speech tags is a crucial step, once new translations are inferred. as this would assist in detecting some ambiguous cases where a miss-match between the parts-of-speech is sufficient to prune them out. the part-of-speech tags could be automatically predicted by taking advantage of neuraland graph-based methods @xcite @xcite . however, in some cases, ensuring the same part-ofspeech tag, might lead to correct translations being filtered out. for instance, the finnish word alla may be an adverb or a postposition, whereas its english translation under is a preposition. in terms of the features used in our neural model, we use the prediction scores returned by the link prediction methods. this causes the neural model to act as an expert voter observing the various scores and to make the executive decision of whether the prediction is valid or not. additional features could be passed to the model, such as the strings of both source and target words, and metainformation about their nodes (e.g., the number of their distinct and common neighbours). based on @xcite work, using the levenshtein distance between source and target words resulted in poor classifications. such features contribute differently to the performance of the model depending on the languages and would limit the model to closely related languages with a high number of cognates. this motivated our choice of judging the quality of predictions based on the link prediction scores, which causes our model to be generic and appropriate for many different language pairs as we assume no phylogenetic relation between the languages in question. this also makes it possible for our approach to work across writing systems as we are dealing with languages written in latin and cyrillic alphabets. ## conclusions and future work we have released the source code of our method and its predictions on github 6 . our method could, in the future, be integrated with the existing dictionary editing infrastructures for uralic languages 6 https://github.com/mokha/translation-link-prediction/ such as giella @xcite and ve'rdd @xcite . this would make link prediction an active part of the process of building lexical resources, making it a more dynamic human-in-theloop task. we have presented our work on extending the existing lexical resources for several endangered languages. for the time being, human annotators are needed to go through the predicted translations, although we have perceived promising results with our neural approach. regardless of the accuracy of the current method for identifying good predictions or what any future method might reach, we believe that a lexicographer needs to go through the predictions at any rate. compiling dictionaries for an endangered language is an important step in the language documentation and, if done right, can greatly benefit the native speakers of the language in learning foreign languages, and also anyone interested in learning the endangered language in question. this being said, any fully automatically produced lexicon will have errors that ultimately lead to misunderstandings and can be harmful for the language community. we envision that our work opens the door for constructing aligned multilingual wordembeddings between endangered languages and high-resource languages. this would narrow the gap between severely scarce-resource languages and the latest neural machine translation techniques, making it possible to build a functional neural translation system from languages at the risk of dying to a vast number of big languages which in return would greatly benefit the communities of endangered language. the results produced by our method will be manually filtered by lexicographers and included in the akusanat online dictionary 7 . the goal of our paper has been that of extending existing lexicographic resources so that the language communities can directly benefit from our research. without releasing our results and having them manually verified, we would be embracing an unethical research tradition that relies on cultural and linguistic appropriation for a purely academic benefit. ment of agence nationale de la recherche as part of the "investissements d'avenir" program, reference anr-19-p3ia-0001 (prairie 3ia institute).