DAG : Dictionary-Augmented Generation for Disambiguation of Sentences in Endangered U ralic Languages using C hat GPT (2024)

Abstract: We showcase that ChatGPT can be used to disambiguate lemmas in two endangered languages ChatGPT is not proficient in, namely Erzya and Skolt Sami. We augment our prompt by providing dictionary translations of the candidate lemmas to a majority language - Finnish in our case. This dictionary augmented generation approach results in 50% accuracy for Skolt Sami and 41% accuracy for Erzya. On a closer inspection, many of the error types were of the kind even an untrained human annotator would make.

## introduction morphological disambiguation is a critical task in natural language processing (nlp), especially for morphologically rich and endangered languages. skolt sami (sms) and erzya (myv), both belong to uralic language family and they are classified as critically and definitely endangered respectively by unesco @xcite . this poses significant challenges in this domain due to their complex morphological systems and limited available linguistic resources (see @xcite . in languages like these, each word form can have multiple possible morphological interpretations and lemmas, and determining the correct one in context is essential for accurate language processing. traditional approaches to morphological disambiguation for uralic languages often rely on finitestate transducers (fsts) and constraint grammars (cgs) that list all potential lemmas for a word, but these systems struggle to accurately select the appropriate lemma in ambiguous contexts -not to mention that cg disambiguators have not even been created to a majority of these languages. additionally, while some modern nlp techniques, such as machine learning models, have been successful in languages with large datasets (see @xcite , such methods are less effective for languages like skolt sami and erzya, which suffer from limited annotated corpora and lexicographical resources. this paper presents a novel method for performing morphological disambiguation for skolt sami and erzya that leverages a combination of a traditional fst-based analyzer, a bilingual dictionary and a state-of-the-art language model, namely chat-gpt. our approach involves passing each sentence through an fst to generate a list of possible lemmas for every word. these lemmas are then translated into a majority language (finnish in our case) using a dictionary. finally, we utilize chatgpt, a powerful transformer-based language model, to analyze the translated sentence, disambiguate the lemmas, and select the most contextually appropriate form for each word. the dictionary needs to be provided given that chatgpt is not proficient in these languages. by integrating the structured linguistic knowledge from fsts with the contextual understanding of large language models, this method aims find a novel way that does not need a time consuming rule-writing or data annotation process for morphological disambiguation for skolt sami and erzya. the proposed approach is particularly valuable for endangered languages, where data scarcity hinders the development of purely data-driven models. this paper details the methodology, presents an evaluation of the approach, and discusses the potential for applying this approach to other morphologically complex languages. we have released our disambiguation code as an addition to uralic-nlp foot_0 . ## related work constraint grammars (cgs) @xcite have been widely used in the context of uralic languages for disambiguation given their compatibility with prompt template actual prompt your task is to disambiguate a sentence in [language] you will be given the sentence, a table that has all of the words of the sentence in separate rows and a comma separated list of possible lemmas. you will need to pick the correct lemma for each word so that every word will have only one lemma. to help you understand [language] you will also get a second table that gives you translations of the words in [language2]. sentence: [sentence] table of lemmas: [table1] [language] -[language2] vocabulary: [table2] please write out the steps of your decision process and provide a list of lemmas in json format at the very end of your answer. example: {"lemmas": ["lemma 1", "lemma 2", "lemma 3"]} your task is to disambiguate a sentence in skolt sami. you will be given the sentence, a table that has all of the words of the sentence in separate rows and a comma separated list of possible lemmas. you will need to pick the correct lemma for each word so that every word will have only one lemma. to help you understand skolt sami you will also get a second table that gives you translations of the words in finnish. sentence: päärna mõ ′ nne mååusat . table of lemmas: ## method the method itself does not require any training or additional annotated data. however, to evaluate our method, we use the universal dependences treebanks for erzya @xcite and skolt sami @xcite . these treebanks have word forms and their correct lemmas for each word in each sentence. given that we do not need to do training, we concatenate the training and test datasets into one dataset for both languages. next, we used the skolt sami @xcite and erzya @xcite fsts through pyhfst @xcite to get all possible lemmas for each word form in the treebanks. for every lemma, we look up its finnish translations from the akusanat dictionary @xcite . as we build our corpus of sentences to be disambiguated this way, we need to do some filtering. if a sentence does not have any ambiguity, all of it's words lemmatized by the fst or all of the potential lemmas mapped to at least one finnish translation, the sentence is removed from the corpus. this way, we end up with 40 sentences from skolt sami treebank and 17 sentences form erzya treebank. the reason why this filtering is done is that chat-gpt has no proficiency in erzya or skolt sami. it does not make sense to try this disambiguation if we cannot provide chatgpt with enough information so that we can even assume that it would be capable of disambiguating the sentence its given. our corpus is used to populate a carefully planned prompt template as seen in the tables are formatted in markdown format. we use openai api and select gpt-4o as the model to be used with temperature set to the default of 1. every sentence is prompted separately so that the results won't have an influence on each other. this experiment cost us $0.41. ## results if we calculate the accuracy of the disambiguation by chatgpt on a sentence level, meaning the number of fully correctly disambiguated sentences out of all the sentences, we get the following results: skolt sami 50% and erzya 41% accuracy. at this step, it is already worth noting that chatgpt omitted some punctuations in its lemmatization. also, skolt sami has several unique unicode characters that look identical to other unicode characters. chatgpt had a tendency of sometimes changing the original characters to their lookalikes in the output. these cases were still counted as correct answers. most of the erroneous sentences had only one word that exhibited ambiguity. in the case of several ambiguous words, chatgpt almost always made a mistake for only one word in the sentence. there were, however, some cases where the same word was repeated twice, in which case chatgpt lemmatized both of them wrong following the same logic. ## conclusions we have demonstrated a viable method for conducting disambiguation on endangered language data. the results are very good given that chat-gpt is not proficient in skolt sami or erzya. if we look at the errors, they mostly make sense to a human. many of the error types are such that even a novice human annotator without training in these languages would make similar mistakes. it is important that we, in the endangered nlp community, keep our eyes and minds open, and embrace the new potential in llms. perhaps they don't speak our languages of interest yet, but they can still make reasoned decisions if enough information is provided to them.