Alligators All Around: Mitigating Lexical Confusion in Low-resource Machine Translation (2025)

Abstract: Current machine translation (MT) systems for low-resource languages have a particular failure mode: When translating words in a given domain, they tend to confuse words within that domain. So, for example, “lion” might be translated as “alligator”, and “orange” might be rendered as “purple.” We propose a recall-based metric for measuring this problem and show that the problem exists in 122 low-resource languages. We then show that this problem can be mitigated by using a large language model (LLM) to post-edit the MT output, specifically by including the entire GATITOS lexicon for the relevant language as a very long context prompt. We show gains in average ChrF score over the set of 122 languages, and we show that the recall score for relevant lexical items also improves. Finally, we demonstrate that a small dedicated MT system with a general-purpose LLM as a post-editor is outperforms a lexicon-based RAG-LLM translator, suggesting a new paradigm for LLM use.

## introduction machine translation systems have recently expanded to cover many previously unsupported languages @xcite . however, mt systems for low-resource languages (lrls) still face many challenges. one particular difficulty is learning the correct mapping of words between two languages. this paper is motivated by the observation that some lrl mt models tend to confuse certain lexical items belonging to similar domains. this problem is first reported in @xcite , who report this issue with unsupervised, sentence-level nmt, giving the following examples from their models. examples from their paper are reproduced in these examples show that the model consistently errs by confusing lexical items that share similar distributions, such as using crocodile to translate other animal terms. this pattern is observed in the "next thousand languages" (ntl) mt models of @xcite over many language pairs and within relatively high-frequency lexical domains, including numbers, colors, animals, days of the week, and months. in this paper, we refer to the tendency to confuse words within a domain as the "alligator problem." foot_0 as we show in this paper, this pattern isn't only found in mt-specific models, but in translations produced by large language models (llms) as well. using a development set consisting of data from 122 lrls, we show that this problem is widespread in translations of the ntl models, which are described in @xcite . we then propose a method for prompting an llm with lexical information to post-edit these translations, both translating into and out of english, leading to better performance on these frequently confused lexical items, as well as higher machine translation quality overall. the lexical information is provided by incorporating the gatitos lexicon @xcite into the llm prompt. we further show that the llm is able to improve its performance on these lexical items even when the lexicon entries presented in the prompt don't exactly match the source string because of morphological inflections. this method combines the in-depth knowledge of the specialist ntl mt systems with the generalist abilities of the llm. we show that the llm is incapable of matching the mt system's performance on its own, even when given access to the lexicon, despite the fact that the mt system is much smaller, at only 850m parameters. however, given the specialist mt model's best hypotheses, the llm can fix the mt model's persistent lexical confusions as a post-editor, making use of the infor-language reference translation meiteilon (mni) i believe a lion is stronger than a tiger. i believe a snake is stronger than a crocodile. twi (ak) i would want to be a dog for a day. i want to be a crocodile just one day. mt for low-resource languages before llms, for very low-resource language mt -i.e. anything beyond the most frequent hundred languages or so -there existed no parallel text at all outside of religious domains. in these cases, the only option was unsupervised machine translation (unmt), which uses only monolingual text to translate. this was pioneered in lample et al. (2017); artetxe et al. (2017); song et al. (2019a), and eventually bapna et al. (2022a) scaled up to 1000 languages in the ntl models. however, the unsupervised paradigm led to tell-tale mistakes, such as the "alligator problem" discussed here. llms then barged in and changed all these paradigms, although they still perform poorly out of the box on lrls @xcite . a common approach is in-context learning, or icl @xcite which gives examples in the prompt. icl examples for lrls have included diverse context like sentence pairs @xcite , dictionaries (elsner and needle, 2023), the full gati-tos lexicon @xcite , and a full grammar of the kalamang language @xcite . a popular variant of icl is rag, or retrievalaugmented generation @xcite , which draws only on examples for icl that are relevant to the current sentence being translated. despite its popularity, @xcite al. (2023); zhu et al. (2024); zhang et al. (2023 ) find exemplar quality is more important than relevance. llms as post-editors. another less common approach for lrl mt has focused on automatic postediting (ape) translations with llms, which is an approach often used in high-resource mt @xcite . chen et al. (2024) let an llm iteratively self-correct its translation, lim et al. (2024) have a model post-edit its own translations from related higher-resource languages into the target language, and @xcite iteratively apply fine-grained error correction from an llm. however, these efforts have focused on a base model and a post-editor that are the same size, and both large. rare word translation many mt models struggle specifically with translating rare words, including mt models for high-resource languages. in our case, we study the inverse problem of difficulties with common words, but the approaches necessary to fix may be the same. prior work includes placing soft constraints on the output terminology (bergmanis and pinnis, 2021) and augmenting parametric models with non-parametric datastores such as parallel corpora @xcite or lexica @xcite . the latter is more similar to our approach, though we present a lexicon to the llm as a part of a prompt, rather than using it during the training phase. ## methods the approach we take to solving this problem is to (1) generate output for a set of lrls using a specialist mt system; (2) create prompts for postediting each segment that include the entire gati-tos lexicon, and (3) use these prompts to generate post-edited output using a generalist llm. the example in ## results and discussion tables 3 and 4 show that the best performance comes from using the llm as a post-editor, and including the entire gatitos lexicon in the prompt. the models we compare are (1), the mt models alone (our baseline), (2) the llm model alone, and (3) the llm as post-editor of the mt model output. the exact prompt templates are in appendix b. the prompts given to the llm include all 4000 entries from gatitos for the given language, except when noted otherwise. as shown in other attempted methods fall short of the performance of llm post-editing with access to the whole lexicon. the llm on its own is a relatively poor translator, even given the entire gatitos lexicon. on these high-confusion languages, we also experiment with presenting the llm with a few different levels of lexical information: no lexical information, prompts with only the words in the given segment, and prompts with the whole lexicon. no lexical information is, as expected, a worse condition, but even limiting the prompt to include only the lexical items that are present in the source is unhelpful -this condition under-performs even the baseline. as expected, the prevalence of lexical confusion correlates with the overall performance of the mt systems on a language, as shown in ## conclusion this work is the first to document and quantify the alligator problem in large language models for low resource languages, a systemic translation error mode that is not well captured in metrics like chrf. this problem is much reduced, though not fully eliminated, by our proposed approach of lexicon-augmented post-editing. this also suggests a new paradigm for generalist models like llms, exploiting their better general-purpose reasoning and tool use to use them as post-editors. the small, specialized mt model provides a strong baseline for translation performance, one that the llm cannot meet on its own, even when given access to a lexicon. however, the llm can better extract and use information from a resource like gatitos, and therefore improve upon its superior's work. the llm is also able to overcome challenges such as complex morphology that would make it prohibitively difficult to use the lexicon directly to post-edit the mt output.