Finding words that aren’t there: Using word embeddings to improve dictionary search for low-resource languages (2023)

Abstract: Modern machine learning techniques have produced many impressive results in language technology, but these techniques generally require an amount of training data that is many orders of magnitude greater than what exists for low-resource languages in general, and endangered ones in particular. However, dictionary definitions in a comparatively much more well-resourced majority language can provide a link between low-resource languages and machine learning models trained on massive amounts of majority-language data. By leveraging a pre-trained English word embedding to compute sentence embeddings for definitions in bilingual dictionaries for four Indigenous languages spoken in North America, Plains Cree (nhiyawwin), Arapaho (Hinno’itit), Northern Haida (Xaad Kl), and Tsuut’ina (Tst’n), we have obtained promising results for dictionary search. Not only are the search results in the majority language of the definitions more relevant, but they can be semantically relevant in ways not achievable with classic information retrieval techniques: users can perform successful searches for words that do not occur at all in the dictionary. These techniques are directly applicable to any bilingual dictionary providing translations between a high- and low-resource language.

## introduction this paper presents an approach for improving the searchability of electronic dictionaries of low-resource languages, exemplified using bilingual dictionaries of plains cree (endonym: nêhiyawêwin; glottocode: plai1258; iso 639-3 code: crk), arapaho (hinóno'éitíit; glottocode: arap1274; iso 639-3 code: arp), northern haida (xaad kíl; glottocode: haida1248; iso 639-3 code: hdn), and tsuut'ina (tsúùt'ínà; glottocode: sars1236; iso 639-3 code: srs), leveraging existing semantic embedding technology for majority languages in the novel context of low-resource minority languages. broadly speaking, search and information retrieval revolves around determining the means by which one may reliably find the most relevant discrete entries (or document(s)) from a set of multiple such documents. in the case of bilingual dictionaries, presenting entry headwords in an minority indigenous language with definitions in a majority target language, the definitions in the majority language (in our case, english) of each entry may be considered the"documents" one searches when using target language (english) search terms. the challenge, therefore, is determining how to find the most relevant indigenous language words (which are the headwords of the entries) for these queries. this is particularly challenging when the sought-after target language definitions do not contain the exact search terms, but instead use related target language words; in these cases, even exact search word matches do not necessarily translate to the highest relevance. for instance, indigenous languages that have a complex morphological system can store large amounts of information and meaning within a single lexeme, which in a morphologically simpler languages (such as english) may need to be represented with multiple words or phrases. consider, for example the plains cree words nôtamiskwêw for 's/he hunts beavers', and êskêw for 's/he makes a hole in the ice to hunt beaver; s/he breaks up a beaver lodge (i.e. in hunting)', which would both require a combination of the english search terms 'hunt' and 'beaver' to be accurately matched. in contrast, an entry such as mâmawohkamâtowak, meaning 'they do things together, they cooperate; they work (at it/him) together as a group; they assemble themselves to help one another.' would be matched with the search terms 'cooperate', or 'work' and 'together', but would be missed with the obvious synonym 'collaborate'. thus, the general problem remains determining the means to capture and represent the underlying meanings of both 1) the targets, the entries represented by their (english) definitions, and in 2) the (english) search terms, particularly when they may be more than the sum of the words in isolation. for endangered languages which are often also lessresourced ones, this challenge becomes greater as the vocabulary contents in their dictionaries (and definitions) are typically much more limited than those in majority languages, resulting in even fewer words to potentially match. for example, even many high-frequency english words, such as national, administration, and network (all within the top 1000 most frequent content words in large corpora such as coca), have no matches whatsoever in the english definitions in any bilingual dictionaries of the four indigenous languages named above. searching with these words using typical methods would therefore result in "no results" -a discouraging outcome for a user -even when these dictionaries actually do contain relevant entries that could be shown. word-embeddings present one possible solution for this. because they represent the underlying concepts that the individual words are pointing at, this allows us to represent concepts, or combinations of concepts, that individual words are pointing at. in turn, this allows for comparing the concepts referred to by the search terms and the entry definitions, rather than the individual words themselves. this paper discusses the implementation and evaluation of this solution to four bilingual dictionaries between an indigenous language and english, all of which we have made available on-line. foot_0 2 background and previous related work ## our approach we will present our approach primarily with examples from itwêwina (itwêwina.altlab.app), an online intelligent bilingual dictionary application, making use of our morphodict platform foot_1 for plains cree -english, although we have implemented this feature also for bilingual on-line dictionaries for arapaho, northern haida, and tsuut'ina, and will present examples from the first two languages in the evaluation section further below. 3 itwêwina is freely accessible to the public and receives roughly 20,000 searches per month. it combines multiple dictionary sources (wolvengrey; maskwachees cultural @xcite , and has approximately 22,000 headwords foot_3 , of which only about 10,000 appear in any plains cree corpus that we know of. through modeling with finite-state transducers (fst) @xcite , it can dynamically recognize wordforms and display paradigm tables for millions of additional inflected word-forms. searches can be entered in either english or plains cree. we break the search process into two phases: retrieval, and ranking. the goal of retrieval is to find potentially relevant definitions for the input query. for cree-language searches, a spell-relaxed finite state transducer identifies potential matching headwords. for english-language searches, the application uses classical information retrieval techniques of matching stemmed keywords between queries and definitions. ranking is necessary because an unsorted list of matches would provide a poor user experience: there may be many hundreds of potentially matching words. therefore, results returned by the finite-state transducer analyzer or classical information retrieval methods are ranked using a combination of result features foot_4 @xcite such as corpus or dictionary frequency, or edit distance. in an attempt to improve the relevance of the top search results returned by itwêwina, we added in spring of 2021 a new result feature to feed into the relevance ranking function: a semantic distance measure, based on word embeddings, between input queries and resultant definitions. while this did improve relevance, the most novel and surprising feature which this revealed was the ability of word embeddings to allow the retrieval of useful search results for words that are not even in the dictionary (target language definitions), in addition to improving the search results for multi-word phrases. ## future work and conclusion one potentially promising research theme to explore is the improvement of multi-word vector creation methods for definitions and search strings, perhaps through the use of termfrequency-inverse-document-frequency weights when adding word vectors to form definition vectors. similarly, investigating whether newer pretrained word embedding models @xcite could produce higherquality results. advances in the use of word embeddings for other nlp tasks in low-resource languages (e.g. @xcite ) may also translate to improved dictionary search.