Translating a low-resource language using GPT -3 and a human-readable dictionary (2023)

Abstract: We investigate how well words in the polysynthetic language Inuktitut can be translated by combining dictionary definitions, without use of a neural machine translation model trained on parallel text. Such a translation system would allow natural language technology to benefit from resources designed for community use in a language revitalization or education program, rather than requiring a separate parallel corpus. We show that the text-to-text generation capabilities of GPT-3 allow it to perform this task with BLEU scores of up to 18.5. We investigate prompting GPT-3 to provide multiple translations, which can help slightly, and providing it with grammar information, which is mostly ineffective. Finally, we test GPT-3’s ability to derive morpheme definitions from whole-word translations, but find this process is prone to errors including hallucinations.

## introduction in low-resource language communities, resource creation efforts are restricted by the limited time community members can contribute-and this problem is worsened when effort must be divided between development of community-facing resources and those targeted at machines. language revitalization and pedagogy programs need dictionaries (especially those which incorporate tools for morphological analysis and flexible search) and grammar lessons, while machine translation systems need large corpora. if community-facing resources could be used within machine learning systems to compensate for the limited availability of text, community efforts could serve both pedagogical and technological goals at the same time. but while this has occasionally been attempted, techniques for doing so are still not effective enough to serve as standard methods. one reason why community-facing resources like bilingual dictionaries have not been widely used in applications like low-resource translation is that understanding definitions can require sophisticated tools for natural language understanding. recent advances in large language model technology (llms) provide a promising candidate for such a toolset, at least for definitions written in highresource languages. in this paper, we investigate methods for using a representative llm, gpt-3, to translate multi-morphemic words in inuktitut using dictionary definitions for the morphemes. in addition, we measure gpt-3's capacity to perform the reverse task of inferring the dictionary definition of a morpheme given a decomposed word in which it occurs. inuktitut is a polysynthetic language in which words can be very long and morphologically complex. as such, it is representative of a number of languages of the americas for which natural language processing tasks have historically been difficult due to limited resources and typological differences from better-resourced languages @xcite . computational tools are an important part of education or revitalization efforts for american languages, including inuktitut @xcite . we show that gpt-3 can stitch together english dictionary definitions to produce reasonable translations of many inuktitut words. we investigate two further questions: methods for dealing with morphemes with multiple definitions, and the extent to which performance can be improved by priming gpt-3 with some grammatical information. we envision our system as one component of an interactive dictionary/translation system, in which a human learner or non-native speaker asks for possible analyses of a morphologically complex form and is given both a morph-by-morph gloss and some possible translations into fluent english. this kind of system might help to bridge the gap between a conventional dictionary, which is incapable of interactively translating morphologically complex words into fluent english, and full-scale neural ma-chine translation (nmt), which is data-hungry and non-transparent. we further see possibilities for suggesting new dictionary definitions which can be curated by native speakers. finally, we believe it holds some potential as a stepping stone towards larger-scale nmt applications by providing simple examples for use in a curriculum learning paradigm @xcite . ## related work the use of bilingual dictionary entries in neural machine translation was pioneered by @xcite , who augment an english/french mt system with mechanisms for aligning rare words across languages, then copying material from definitions to translate them. this work makes several key assumptions about the benefits dictionary definitions can provide. in particular, it assumes a relatively capable nmt system already exists. because of this, the main contribution of the dictionary is to provide lexical equivalents for rare items. in most cases, these are content words, and their definitions, once known, are easily integrated into the translated sentence @xcite . @xcite ; niehues (2021) are among the earliest to provide dictionary information as augmented input rather than via a custom architecture (thus moving toward a zero-shot method), but still requires the nmt system to be trained to use definitions. our work follows from recent approaches which use llms rather than purpose-built nmt systems. the ability of llms to translate some highresource language pairs in a prompt-based zero or few-shot setting is established by @xcite . some other recent papers attempt to augment llms with information derived from dictionaries or phrase tables. @xcite translates full sentences, using hints from a phrase aligner @xcite . their approach also emphasizes the monolingual text-to-text generation capacity of llms, but uses aligned phrases rather than dictionary definitions intended for human readers. the closest point of comparison to our work is @xcite , which improves translation performance by augmenting few-shot prompts with dictionary translations for a few selected words. unlike our setting, where the baseline nmt system performs poorly, their languages are selected so that the baseline nmt model per-forms reasonably without augmentation (10-30% bleu). in contrast to these approaches, we assume a setting in which common morphs, including functional as well as content items, must be translated with the aid of the dictionary. while this is an artificial constraint in the case of inuktitut, which does have enough parallel text to train an nmt system, it is the case for other low-resource polysynthetic languages of the americas which lack parallel corpora large enough to train any nmt system. even where data is more plentiful, dictionary definitions are potentially helpful for translating functional items in polysynthetic languages because these languages can have very large paradigms with very unbalanced attestations in corpora. inuktitut has a polypersonal agreement system in which subject and object person and number are both marked on verbs; some subject-object markers rarely appear in written corpora due to discourse constraints (for example, dual subjects). other morphosemantic distinctions (such as dual number, evidentiality, intensifiers and applicatives) which are common in american languages but rare in european ones, are translated in very different ways across contexts, leading mt systems to misalign them due to limited data @xcite . in this setting, dictionary definitions may be patched together in relatively complex ways. first, composing the definitions is more difficult than simple concatenation: takujara "i see him/her" is made up of taku "see" and jara "i . . . him/her" (1sg>3sg). second, inuktitut uses derivational processes to express terms which have independent content words in english. qukiut "gun" is made up of qukit "shoot" and the instrumental marker ut. while a literal translation would produce "an instrument for shooting", the leap to paraphrase this expression as "gun" requires a deeper representation of english semantics. thus, while previous systems could use dictionary material mainly by copying, our task setting emphasizes text-to-text generation. related tasks which use language modeling to patch together fragments of target-language structure include bag-to-sequence word ordering @xcite and dependency linearization @xcite . generating fluent text from grammatical element annotations is also similar to generating translations from glosses @xcite , although, despite re-cent interest in glosses (e.g. @xcite , this task is also comparatively understudied. inuktitut itself is one of the best-resourced indigenous american languages, with a large parallel corpus collected from the nunavut parliamentary hansards @xcite ; their baseline nmt system yields an @xmath0.0. this dataset was used as a challenge for the workshop on machine translation in 2020 @xcite . scores remained low compared to better-resourced languages-the system rated highest by humans reports a bleu score of only 29.1 on test @xcite . as stated, we use inuktitut as a potential model for less-resourced polysynthetic languages where even this level of nmt performance is not available. in addition to assuming access to a dictionary, we also assume access to a system which provides canonical morphological segmentations (based on @xcite and further described in section 3), so that a complex word can be decomposed into parts whose lexical entries can be accessed. such systems @xcite can be developed with access to substantially less data than nmt systems. they may be created using finitestate toolkits @xcite or supervised learning from relatively small annotated datasets @xcite , potentially incorporating active learning @xcite . segmentation of inuktitut is easier than translation-uqailaut @xcite is a widely used finite-state system for canonical segmentation of inuktitut. @xcite describes an improved segmenter with neural disambiguation; @xcite conduct more recent experiments on segmentation using transformer networks. while our assumption that a segmentation system is available constitutes a weakness of our method, we hope that future work can continue to reduce the resource burden of developing such systems and can also tie their development more closely to communityfacing resources such as grammar texts. ## data and metrics we extract a lexicon of inuktitut morphemes and their definitions from the uqausiit dictionary (uqausiit.ca) created by inuit uqausinginnik taiguusiliuqtiit, an inuit language authority funded by the nunavut legislature. uqausiit also provides example phrases with english translations and hand-annotated partial morphological decom-positions. we extract all single-word example phrases with a multi-morphemic partial decomposition for development and testing of our translation systems. tables 1 , 2 and because our aim is primarily to evaluate the potential of llms to operate on dictionary definitions, rather than to evaluate algorithms for segmentation, we use a fixed canonical morphological segmentation for each word in our dataset. these are provided by a partial oracle which is implementationally simple to create. we run the uqailaut fst segmenter @xcite , which produces a set of candidate analyses. we then intersect these analyses with the partial decomposition of the phrase from uqausiit. if one or more analyses match, we select the first one and return it. if uqailaut cannot analyze the word, or produces no analyses matching the partial decomposition, we do not use the word. (many of these errors result from orthographic or dialectal variation beyond the scope of uqailaut's design, as noted by @xcite .) for instance, sikujuittuq "an area of the ocean where ice does not form" has the uqausiit partial segmentation siku-juit, without the final tuq. uqailaut produces 6 candidate segmentations, which vary as to the analysis of the medial juitt sequence; of these, we select siku-juit-juq since this matches the provided partial segmentation. (the resulting segmentation may still contain an incorrect element which is not part of the partial decomposition; we have no way to measure how often this occurs, but did not find cases in development.) we also create datasets for creating dictionary definitions, for both root and functional morphemes. in each case, we use a whole word with its translation as the prompt, and query the definition of one of the component morphemes. we assume the root morpheme is the first one in the word; for functional morphemes, we select a later one at random. we create instances for all 219 segmentable words in the test set. foot_0 some of these instances ask for a definition of the same query morpheme (but with different whole-word prompts); there are 89 unique root types and 130 unique functional types in the dataset. where a query morpheme has multiple possible definitions, we refer to uqausiit and the whole-word definition to select the correct one as a reference. we evaluate translations against the english references using sacrebleu @xcite and bleurt @xcite foot_1 . in some cases, we ask the system to produce multiple candidate translations. we anticipate a human user of the dictionary considering the context in which they encountered the phrase they are looking up and picking the best one; this may be especially helpful in cases where the phrase is actually ambiguous in its translation, since otherwise there is no way to pick between candidate meanings. for these, we evaluate bleurt in two ways: the average performance is the expected score of each response against the reference, and reflects the user's experience if they are looking up a word for which they have no useful context. the oracle score is the best score of any response, and reflects the user's experience if they can always pick the correct translation given the context in which they heard the target word. because sacrebleu produces a global precision score over the whole corpus, we evaluate the expected sacrebleu in multiple-translation cases by sampling one translation from the set of proposals for each word; we average across five samples. we produce an oracle sacrebleu score by collating the translations of each word with the highest bleurt score and evaluating them as a group. the inuktitut word saviggirunnaqtutit is made up of the following parts: savik: (1) metal; steel; iron (2) snow-knife; or, to be fitted with a metal point (harpoon; spear) ggiq: to bring someone or something along, as in "don't bring your gun along." runnaq: to be able to perform a certain action, as in "could you find that out if he/she arrived" tutit: you, as in "you sleep" in english, saviggirunnaqtutit means roughly: figure 1 : an example prompt for the defini-tion+example case. ## single-word translation all our experiments use the openai api to access the gpt-3 model text-davinci-003, which was the largest gpt-3 available until the release of gpt-3.5-turbo in march 2023. we do not experiment with models tuned for chat using reinforcement learning, nor with smaller but more efficient llms such as t5. because the goal of this work is to generate translations, rather than to measure the acceptability of pre-existing translations, we sample strings from the model rather than measuring their probability @xcite ; we use the standard text completion api with temperature .7 and 128 maximum tokens. to translate an inuktitut word into an english phrase, we look up every morph in the canonical segmentation and obtain their dictionary definitions. we then produce one or more prompts for the gpt-3 system. figure 1 shows a sample prompt for the definition+example method; examples of the other prompts are given in appendix a. concatenate: as a trivial baseline, we simply concatenate the morpheme definitions in order, without using the llm. this provides a point of comparison for evaluating the improvements due to text-to-text generation. our next two methods evaluate the usefulness of specific parts of the dictionary entries. in the definition only condition, we provide textual definitions of each morpheme; if a morpheme matches multiple dictionary entries, we concatenate them with "; or, " as the separator. in the definition+example condition, we also provide the english translation of an example word in which the morpheme is used (if the dictionary contains one). we evaluate two more sophisticated methods for dealing with morphemes with multiple possible meanings. first, we ask gpt-3 textually to list all possible meanings for the word, rather than producing only a single one; we call this setting multianswer. second, we enumerate all combinations of morpheme meanings which could make up the word, and create a separate prompt for each one. (this method requires much more computing time than simply prompting for more than one answer.) we call this setting multiprompt. in each case, the separate answers are aggregated in two different ways-average and oracle performanceas described in section 3. we consider two methods for injecting grammatical information into gpt-3's processing. first, we preface the morpheme decomposition with a short hand-written grammar description. our grammatical description is intended to focus gpt-3 on some common issues we noticed in development. it explains that inuktitut words begin with a root morpheme which usually determines the syntactic type of the word. verbs must be translated as english sentences whose subject and object are given by agreement markers at the end of the word, while nouns must be translated as noun phrases or prepositional phrases. we also explain that intermediate morphemes can change the part of speech and contribute other elements to the meaning. we also experiment with a chain-of-thought method in which the system is instructed to explicitly identify the syntactic category of the root, the category of the target translation, the subject and object (if any) from agreement morphology, and the function of any intermediate morphemes before translating. we evaluate all the prompts, except the chainof-thought and multiprompt methods, in both zeroshot and few-shot settings. the chain-of-thought method is used only in few-shot mode, since this allows us to model what the intermediate reasoning steps should look like. the multiprompt method is used only in zero-shot mode as, since a prompt is generated for each combination of definitions, it is extremely expensive to run with longer prompts. our few-shot prompts are always filled in with a pre-selected list of the same five words, with definitions and grammatical decompositions from an inuktitut pedagogy site, tusaalanga.ca. three of these are translated as sentences, one as a noun phrase and one as a locative prepositional phrase. two of the sentences have intransitive subject agreement markers and one has transitive subject-object agreement. we fill out the possible answers in the multi-answer condition and the chain of thought reasoning steps manually based on tusaalanga. ## definition creation we experiment with a single prompt for eliciting definitions. this prompt (figure 2 ) provides the definition of all but a single morpheme, and the translation of the phrase as a whole, then asks for the definition of the missing item. gpt-3 is somewhat less likely to restrict itself to the prompted format in this case. we cut off elicited definitions at the first newline. for 20 of the roots, and 3 functional morphemes, gpt-3 repeats the prompt phrase "currently unknown." inspection of the definitions ( on the other hand, the definitions of qingaq "nose" in #2 and kisaq "to anchor" in #3 are motivated by the provided examples, but too specific. kisarvik (#3), for instance, is made up of the target morpheme kisaq and vik, which creates a place nominal from a verb. the system therefore should infer that the target morpheme is a verb and does not contribute the meaning element "place". instead, it proposes the nominal meaning "an anchorage or place to tie up a boat." the system also hallucinates some entirely unmotivated definitions, such as "bay, inlet or cove" for vik. this definition is likely taken from the system's knowledge of a different language such as norwegian or swedish, in which vik has this meaning. the extra work of filtering out these misleading definitions detracts from any benefit of using the system for dictionary expansion, since more time may be wasted than gained. these proposed definitions are also not suitable for presentation to a language learner, who may be misled by their seeming plausibility. ## conclusion our results clearly establish that gpt-3's capacity for monolingual text understanding and generation are sophisticated enough to allow it to translate some complex words with the aid of a dictionary alone. investigation of various prompt tuning tech-niques leads to a few tentative conclusions. usage examples from the dictionary are helpful in combination with definitions. so are few-shot examples (even when only 5 are available). we investigated asking gpt-3 to generate multiple translations, and found some evidence that it might be helpful, particularly when the user already has some idea of how the word might be disambiguated based on the context in which it was encountered. experiments on better-resourced agglutinative languages, such as finnish and hungarian, would be useful in better differentiating the performance of different prompting techniques. on the other hand, we were not able to show much benefit from adding information on grammar to the prompts. there are two possible reasons: perhaps we engineered the prompts poorly and could have obtained better results with more careful decisions about what information to include. or perhaps the gpt-3 system is not yet sophisticated enough to relate relatively abstract grammar information to the concrete task of translation. we believe both of these explanations are valid to some extent. evidence that our prompt engineering is at fault comes from examining the failures of the chain-of-thought system in more detail (appendix b). as stated above, the system sometimes misses nominalizers and verbalizers; although the prompt states that these exist, we could have included more detailed information on how to recognize them. on the other hand, the system also sometimes creates complex chains of thought in which it loses track of content morphemes. this is why it predicts the meaning "you can make it" for #1 in while better prompting could potentially remedy some of these issues, we believe more sophisticated instruction-following models may perform even better given the same resources. moreover, models with very long context windows might be able to read in large sections of pedagogical material from sites like tusaalanga directly, reducing the necessity to create abbreviated grammar lessons specifically for use in prompting llms. we are also interested to see to what extent more sophisticated llms can improve at suggesting dictionary definitions, especially to the extent that hallucinations can be controlled. on the other hand, it would be potentially interesting to see whether these results can be replicated with smaller, more cost and computation-effective llms such as t5. while this work emphasizes how much is possible without an nmt system, we also believe that it can contribute to nmt development in the very low-resource case. curriculum learning for translation (e.g. platanios et al., 2019) uses translations of shorter or simpler constructions earlier in pretraining, but suitable "easy" instances may be rare in corpus data, or their distribution may be skewed towards formulaic language. few-shot dictionarybased translation could be used to bootstrap towards a larger nmt system by providing candidate definitions for single words from the corpus. although the scope of the present work is limited to an exploratory demonstration, we are eager to see the many ways in which it can be expanded upon. one particular direction is to explore the extent to which an llm can exploit linguistic context to disambiguate between various potential translations, hopefully leading to a narrowing of the gap between average and oracle performances. more closely integrating segmentation into the prompting system, either by having the llm produce its own segmentations or rank multiple segmentations based on the plausibility of their meanings, would reduce dependence on an accurate canonical segmentation system. we are (to our knowledge) the first to evaluate dictionary-based translation in the absence of a base nmt system, and the first to deploy it on a polysynthetic language. while our results are not yet competitive with fully trained translation, we believe our results represent good news for communities in which limited resources must be distributed among efforts to develop communityfacing resources or parallel corpora. a community that focuses its effort on developing dictionaries for human learners can nonetheless enjoy some of the benefits of mt without developing a conventional nmt system, helping to bring language revitalization and language technologies closer together.