LLM -Assisted Rule Based Machine Translation for Low/No-Resource Languages (2024)

Abstract: We propose a new paradigm for machine translation that is particularly useful for no-resource languages (those without any publicly available bilingual or monolingual corpora): LLM-RBMT (LLM-Assisted Rule Based Machine Translation). Using the LLM-RBMT paradigm, we design the first language education/revitalization-oriented machine translator for Owens Valley Paiute (OVP), a critically endangered Indigenous American language for which there is virtually no publicly available data. We present a detailed evaluation of the translator’s components: a rule-based sentence builder, an OVP to English translator, and an English to OVP translator. We also discuss the potential of the paradigm, its limitations, and the many avenues for future research that it opens up.

## introduction large language models like openai's gpt series @xcite have shown remarkable capability at an impressively wide range of natural language tasks @xcite including machine translation @xcite . these models work because they are trained on vast amounts of natural language data, primarily from the internet @xcite . for languages that are lowresource (languages for which there is little publicly available data) or no-resource (languages for which there is no publicly available data), models like these do not perform well on their own @xcite . there have been many efforts in improving machine translation for low-resource languages (see @xcite for a comprehensive survey), but no-resource languages have received much less attention in the literature. over past decades, researchers and community leaders have led many remarkable efforts in endangered language revitalization @xcite and reclamation @xcite . in this paper, we begin to explore how the impressive general-purpose language skills of llms might be helpful for these kinds of efforts by introducing a new paradigm for low/no-resource machine translation: llm-rbmt (llm-assisted rule-based machine translation). the intuition behind our approach is that, given the correct tools, humans are good at translating simple sentences even for languages they do not know. for example, a common task for students in a language class is to translate sentences given a set of words and rules. given a conjugation table and a set of verbs a motivated student could probably translate a list of simple subject-verb (e.g., "i eat", "you sing", etc.) sentences with extremely high accuracy. of course, they are limited to only translating specific kinds of sentences with a limited vocabulary. still, the idea is interesting: if we provide enough context to an llm like chat-gpt, which has been shown to exhibit human-level performance on many natural language tasks, we may not need it to actually know (exhibit general fluency in) the target language we want to translate to/from. while rule-based translation can likely never achieve the quality of modern ml translators for high-resource languages, this technique has a few potential advantages for low/no-resource languages. first and most importantly, it requires no parallel corpus. also, it is amenable to partial translation. humans (and, as we'll show in this paper, llms) are capable of giving partial translations when they don't have all of the necessary vocabulary. for example, a spanish student that doesn't know the word for "dog" might still be able to translate the rest of the sentence "the dog ate the apple yesterday" by saying "el [dog] comió la manzana ayer". in this paper, we leverage llms (specifically, gpt-3.5-turbo and gpt-4 from openai's gpt series) to break down natural language sentences into structured simple sentences compatible with handcrafted rule-based translators. we also use them to turn sentence structured information (in json format) into natural language sentences. using this approach, the llm never interacts directly with the target language. rather, we rely on the llm to tell us how to use the simple, rule-based translators to provide a translation as close as possible to the user's original input. we use this technique to build and evaluate two machine translation tools for owens valley paiute (ovp, also called eastern mono or monache in linguistics literature, iso 639-3 mnr (sil international, 2024)), a critically endangered indigenous american language in the uto-aztecan language family @xcite . the first is a selection-based ovp to english simple sentence translator and the second is an english to ovp translator that uses available vocabulary to construct translations of arbitrary user input sentences. the translators are oriented toward language teaching and revitalization. they are not designed to be general-purpose translators, but rather as tools to help no-resource language learners express ideas using simple sentence constructions. ## related work the landscape of low-resource machine translation is vast and constantly growing. a comprehensive survey on this subject is provided by @xcite , which outlines the current techniques, guidance for selecting which techniques to use for a given language, and future directions for research. of particular interest within this survey is the examination of unsupervised neural machine translation. while there exists promising research on constructing translators from minimal corpora, these methods invariably require some natural language data and thus have limited applicability to noresource languages (such as ovp). the survey also discusses data augmentation strategies, including word or phrase replacement-based augmentation and back-translation-based data augmentation, both of which could potentially be integrated with some of the solutions presented in this paper (in particular the sentence builder and ovp to english translator to be presented in section 3). such an exploration is an interesting topic for future work. other approaches discussed in the survey such as supervised learning, transfer learning, and semisupervised learning are inapplicable to our scenario due to the absence of bilingual or monolingual corpora. contrary to the prevailing assumption in the literature that rule-based machine translation (rbmt) is a relic of the past, there remains active research and development in rbmt systems tailored for the most under resourced of languages @xcite @xcite . recent work has also explored the utilization of large language models (llms) for enhancing translation capabilities in low-resource languages through fine-tuning techniques @xcite . although this approach has shown promise in improving llm translation quality for low-resource languages like (e.g., irish), its reliance on bilingual corpora make it infeasible for no-resource languages like ovp. recently, semantic similarity has been used to evaluate the quality of machine translation systems @xcite . we observe that this technique is particularly useful for evaluating the quality of the english to ovp translator presented in this paper, due to the lack of parallel corpora. see @xcite for a comprehensive benchmark of different embeddings models used for computing semantic similarities. ## ovp to english translation in this section, we present an llm-assisted selection-based ovp to english translator. the first piece of this translation system is a selectionbased ovp sentence builder. the sentence builder allows users to select each of the following parts of speech from a list of choices until they form a valid sentence: not all of these parts of speech are required to create a valid sentence in ovp, though. in fact, some of them are incompatible. for example, if an intransitive verb is chosen, then it cannot have an object or object pronoun. in other words, the valid choices a user can make is a function of the choices they have already made. in our python implementation for this translator, we process each user selection and change the list of valid options for each part-ofspeech based on their current selections to ensure they always create a valid ovp sentence. this is the rule part of llm-assisted rule-based machine translation that requires expert knowledge in the target language to implement. the entire vocabulary available for the users to select among fits on a single page (and can be found in appendix c). after the user creates a valid ovp sentence, we translate it by first encoding the following sentence information into an english-only (using vocabulary definitions) structured simple sentence: to evaluate the accuracy of the translator, we generated 100 random valid ovp sentences by iteratively selecting a random choice among available choices for each of the parts of speech until the sentence is valid. of the 100 random sentences generated, 98 were translated into english accurately using gpt-3.5-turbo model from openai's gpt-series. translations and their accuracy labels can be found in appendix a. while impressively accurate, this translator has many disadvantages. it only works for simple subject-verb and subjectverb-object sentences that use the nouns and verbs available in the system. also, since pronouns and suffixes in ovp encode temporal/spatial information, translations don't always capture full meaning of the sentence. the english translations are correct, but may be missing useful information. for example "kamü-uu wo'abi-neika a-düka-ti" translates to "the jackrabbit is eating the worm", which is technically correct, but -uu also indicates that the jackrabbit is not present and the -neika indicates the worm is present. then, since -ti can is used for both the present and past continuous tenses (is wo'ada-ii pagwi-noka u-zawa-dü. despite some expected disadvantages, this translator has many advantages. first, it is the first machine translator for ovp. it is also easy to extend the tool with new nouns and verbs. also, while implementing the rules required expert knowledge of what makes an ovp sentence valid, no expert knowledge of how the rules map to english was required (or needed to be programmed), thanks to the llm. finally, we believe this kind of translation system might be a useful educational tool that helps students learn how to build simple sentences. it also may be useful as a data augmentation technique for training neural machine translation models for low-resource languages. ## english to ovp translation in this section, we present the first english to owens valley paiute machine translator. unlike the ovp to english translator presented in the previous section, the user can input any sentence in natural language. the translator works by first using an llm to break the input sentence into a set of simple structured subject-verb and subject-verb-object sentences, discarding any adjectives, adverbs, prepositions, objects (except for direct objects), etc. we encourage (through few-shot prompt engineering) the llm to preserve as much semantic meaning as possible between the original sentence and the set of simple sentences. consider the example in figure 2 where few-shot examples (colored black) tell the llm how to respond to a given input sentence "we are playing and laughing." (colored blue) foot_1 . then, we use these structured sentences and available vocabulary to build valid ovp sentences with the sentence-building tool described in section 3. once the sentence is built, we use the translator described in section 3 to translate the ovp sentences back into english. the idea is that, while some meaning may have been lost between the original input sentence and the final output english sentences, the user can be fairly confident (given the accuracy of the ovp to english translator) that the final translations are correct. the entire english to ovp translation process is depicted in figure 3 . we are playing and laughing. we evaluate the system by translating a set of 125 sentences. there are five types of sentences in the dataset (25 of each): or "john read a book") • two-verb (e.g., "she sings and dances." or "i ate while watching tv.") • two-clause (e.g., "my brother drove and i waited." or "harry wrote and ron read.") we translated all 125 sentences using both the gpt-3.5-turbo and gpt-4 models from openai's gptseries, resulting in a total of 250 translations. to measure the quality of each translation, we compute the semantic similarity between the input sentence and: the usefulness of these measurements depends greatly on the function used to compute semantic similarity. we compute semantic similarity by generating embeddings for sentences (using some embeddings model) and computing the normalized cosine similarity between these embeddings. for our application, we want semantically similar sentences to have small normalized cosine similarity, independent of other linguistic features like syntax. for example, an ideal semantic similarity function would rank "an apple is eaten by a man" more similar to "a man eats apples" than the sentence "a woman drinks coffee", despite the latter sentence being essentially grammatically equivalent to the target sentence. we evaluated seven different embeddings models for this purpose and measured the semantic similarity between twelve target sentences and a ranked list of 10 sentences for each ranging from most to least semantically similar (sentences can be found in appendix b). for each target sentence, we compare the ground-truth ranking of the 10 sentences to the ranking determined by the semantic similarity scores yielded by a particular embeddings model. we measure the similarity between the two rankings using two metrics: average displacement (average distance between a sentence's position in the computed ranking and its position in the target ranking) and @xcite ). we computed the semantic similarity between all pairs of sentences in the dataset to establish a baseline for comparison. the mean semantic similarity between a pair of unrelated sentences was µ ≈ 0.574 with a standard deviation of σ ≈ 0.061. furthermore, the distribution appears to be relatively gaussian (a histogram can be found in appendix e). intuitively, this suggests that semantically unrelated sentences are very unlikely to have a semantic similarity score of above µ + 3σ (i.e., greater than 0.757). a good translation, then, should score high on all three semantic similarity metrics. for example, the translation i am swimming. simple semantic similarity i am swimming. 1 comparator i am swimming. 1 target nüü pahabi-ti. backwards i am swimming. 1 (model: gpt-3.5-turbo) simple semantic similarity brother went. i went. 0.794 comparator [subject] went. i went. 0.608 target mia-ku [brother]-uu. nüü mia-ku. backwards the brother went. i went. 0.806 (model: gpt-3.5-turbo) another interesting case is when the simple and comparator scores are high and only the backwards score is lower. this is observed in cases where there is ambiguity in ovp where there is not in english. for example, in the translation she is cooking. ## conclusion in this paper, we present the first translation tool for owens valley paiute (a critically endangered indigenous american language) and, in doing so, propose a new methodology for low/no-resource machine translation: llm-rbmt (llm-assisted rule-based machine translation). due to a lack of bilingual or monolingual corpora, we use semantic 3 the cost to run generate all translations was @xmath05.48 using gpt-4. similarity to measure the quality of translations, and observe interesting cases where the translator performs well, where it performs well given the vocabulary (primarily verbs and nouns) available to it, and where it performs poorly due to current implementation limitations. this research opens up many directions for future work. first, expanding the translation tool (e.g., by adding more vocabulary, building versions for other languages, etc.) will make it more useful for the communities it is meant to serve. we are currently working to add more complex sentence structures to test the limits of this methodology. building this system required expert knowledge of the target language. it may be interesting to explore how large language models might be able to help with this task as well (e.g., in finding patterns and implementing rbmt logic) to make it easier for non-experts (and even experts in the language who are less comfortable writing code) to build these kinds of translators. finally, other methodologies for leveraging llms for low/no-resource machine translation should be explored. for example, a rag (retrieval augmented generation) @xcite approach that searches a knowledgebase for known sentence structures, vocabulary, grammar rules, etc. and uses them to perform zeroshot translation might work well. the remarkable general-purpose language skills that llms exhibit make them a promising tool in helping revitalize critically endangered languages. t h e b ir d c h a s e d a w o r m . t h e b e a r c a u g h t a fi s h . t o m c a u g h t a fi s h . r a c h e l lo v e s c o ff e e . t h e s u n h e a t s t h e e a r t h . t h e d o g a t e s o u p . s u s a n s in g s s o n g s . r o m e o w r o t e a le t t e r . j o h n r e a d a b o o k . t h e t e a c h e r g a v e a le c t u r e . t h e c a p t a in s t e e r e d t h e s h ip . m ik e w a t c h e d a m o v ie . h a r r y r e c e iv e d a le t t e r . j a c k f o u n d a p a il . p a u l p la y s t h e g u it a r . t h e w r it e r fi n is h e d a s t o r y . m o m m a d e d in n e r . t h e li o n h u n t e d a d e e r . a n n a b a k e d a c a k e . t h e k in g w o r e a c r o w n . b a t m a n c a u g h t a t h ie f . t h e c h e f p r e p a r e d a m e a l. a li c e o p e n e d a d o o r . l in d a b o u g h t a d r e s s . m y b r o t h e r d r iv e s a c a r . 0 0.2 0.4 0.6 0.8 1 generated sentence, model simple, gpt-3.5-turbo simple, gpt-4 comparator, gpt-3.5-turbo comparator, gpt-4 backwards, gpt-3.5-turbo backwards, gpt-4 sentence similarity sentence @xmath1. w e a r e p la y in g a n d la u g h in g . i w il l c o o k a n d e a t . s h e s in g s a n d d a n c e s . t h e y a r e c li m b in g a n d e x p lo r in g . w e w il l c e le b r a t e a n d d a n c e . i r e a d a n d w r it e . h e v is it e d a n d e n jo y e d . b ir d s w e r e c h ir p in g a n d fl y in g . i a m s w im m in g a n d r e la x in g . t h e y w e r e la u g h in g a n d jo k in g . s h e s m il e d a n d w a v e d . i w a s r u n n in g a n d li s t e n in g t o m u s ic . b ir d s w il l m ig r a t e a n d r e t u r n . w e t r a v e le d a n d d is c o v e r e d . t h e y w il l t r a v e l a n d e x p lo r e . h e w o r k s a n d s t u d ie s . s h e is c o o k in g a n d t a s t in g . i a t e w h il e w a t c h in s h e w il l s t u d y a n d p a s s . t h e b ir d c h ir p e d a n d t h e s q u ir r e l ju m p e d . t h e d o g a t e s o u p a n d t h e c a t r a n . s u s a n s a n g a n d i r e a d . m ik e c o o k e d a n d h is s is t e r d a n c e d . r a c h e l d a n c e d a n d m o n ic a s a n g . t h e b e a r r o a r e d a n d t h e f o x h id . t o m s le p t a n d j e r r y w a t c h e d . h a r r y w r o t e a n d r o n r e a d . j a c k c li m b e d a n d j il l f o ll o w e d . t h e k in g s a t a n d t h e q u e e n s t o o d . a li c e la u g h e d a n d b o b c r ie d . m o m la u g h e d a n d d a d c r ie d . a n n a s w a m a n d e ls a s k a t e d . t h e t e a c h e r s p o k e a n d t h e s t u d e n t li s t e n e d . t h e w r it e r t y p e d a n d t h e p o e t t h o u g h t . r o m e o c a ll e d a n d j u li e t a n s w e r e d . t h e c a p t a in s h o u t e d a n d t h e c r e w r e s p o n d e d . t h e c h e f b a k e d a n d h is a s s is t a n t c le a n e d . p a u l ju m p e d a n d p e t e r d u c k e d . b a t m a n s w u n g a n d r o b in la n d e d . t h e s u n s h o n e a n d t h e m o o n g lo w e d . t h e li o n h u n t e d a n d t h e t ig e r r e s t e d . j o h n t r a v e le d a n d m a r y s t u d ie d . l in d a p a in t e d a n d h e r f r ie n d s k e t c h e d . m y b r o t h e r d r o v e a n d i w a it e d . 0 0.2 0.4 0.6 0.8 1 generated sentence, model simple, gpt-3.5-turbo simple, gpt-4 comparator, gpt-3.5-turbo comparator, gpt-4 backwards, gpt-3.5-turbo backwards, gpt-4 sentence similarity sentence @xmath2. t h e b e a r a n d t h e f o x f o r a g e d f o r f o o d . l in d a a n d h e r f r ie n d d a n c e g r a c e f u ll y . t h e d o g a n d t h e c a t w e r e e a t in g s o u p . s u s a n a n d i w il l w a t c h a m o v ie . t h e c h e f a n d h is a s s is t a n t p r e p a r e d a f e a s t . p a u l a n d p e t e r p la y t h e g u it a r . t h e w r it e r a n d t h e p o e t c o ll a b o r a t e d o n a b o o k . t h e c a p t a in a n d t h e c r e w s a il e d t h e s e a s . t h e t e a c h e r a n d t h e s t u d e n t w e r e d is c u s s in g . b a t m a n a n d r o b in f o u g h t c r im e . t h e li o n a n d t h e t ig e r r o a r e d . t o m a n d j e r r y c h a s e e a c h o t h e r . a li c e a n d b o b s o lv e p u z z le s . a n n a a n d e ls a s h a r e a b o n d . t h e k in g a n d t h e q u e e n r u le d w is e ly . m ik e a n d h is s is t e r s t u d y t o g e t h e r . t h e s u n a n d t h e m o o n a r e c e le s t ia l b o d ie s . r a c h e l a n d m o n ic a s h a r e a n a p a r t m e n t . r o m e o a n d j u li e t lo v e d d e e p ly . j o h n a n d m a r y a r e t r a v e li n g . m o m a n d d a d a r e c e le b r a t in g t h e ir a n n iv e r s a r y .