Using Language Models to Disambiguate Lexical Choices in Translation (2024)

Abstract: In translation, a concept represented by a single word in a source language can have multiple variations in a target language. The task of lexical selection requires using context to identify which variation is most appropriate for a source text. We work with native speakers of nine languages to create DTAiLS, a dataset of 1,377 sentence pairs that exhibit cross-lingual concept variation when translating from English. We evaluate recent LLMs and neural machine translation systems on DTAiLS, with the best-performing model, GPT-4, achieving from 67 to 85% accuracy across languages. Finally, we use language models to generate English rules describing target-language concept variations. Providing weaker models with high-quality lexical rules improves accuracy substantially, in some cases reaching or outperforming GPT-4.

## introduction resolving ambiguity in translation is a fundamental challenge @xcite that remains unsolved @xcite . this paper focuses on lexical selection, a key aspect of translation that requires using context in a source sentence to determine the best translation for an ambiguous source word from several target-language options. figure 1 shows variations of the concept date (fruit) in farsi and an example of the lexical selection task. our work has two main goals. first, we investigate the capabilities of language models in disambiguating lexical choices in translation by comparing instruction-tuned language models with high-performing neural machine translation systems. second, we test whether language models can be used to extract useful natural language rules that accurately describe how to translate ambiguous words based on source-side context. we work with native speakers to introduce the dataset of translations with ambiguity in lexical selection (dtails), a test set of 1,377 sentence pairs spanning nine languages where concept variation can be explained by context in the source sentence. evaluating five models on dtails reveals that, without rules provided in-context, only the best-performing llm, gpt-4, is competitive with nmt systems. we also present a simple method for generating rules for lexical selection from llms, which native speakers verify are highly accurate. figure 1 shows rules generated for three farsi variations of the concept date. we observe improvements in performance across all llms when prompted with accurate self-generated rules. in addition, while open-weight llms lag behind both nmt systems and gpt-4, providing rules from gpt-4 can help substantially to bridge the gap in performance. this suggests that parametric knowledge of concept variation poses a greater challenge to models than the ability to apply such knowledge in-context. our work demonstrates that lms can generate highquality rules, and further leverage these rules to rival specialized nmt systems on lexical selection, but still fall short of native speakers. 1 2 task and data to evaluate a model's ability to understand concept variations, we study lexical selection in translation. for example, the noun date has multiple lexical variations in farsi, which distinguishes variations by fruit ripeness and dryness (figure 1 ). we collect a dataset of translation pairs that require understanding and appropriately applying concept variation, where sufficient context is provided to distinguish between variations. ## rules for lexical selection we experiment with generating human-readable english rules that describe all extracted concepts and their variations, and analyze how these rules influence model performance on lexical selection when provided as input. 9 for each target-language variation, we find the 50 longest source-language sentences with less than 50 tokens in length where the variation appears in the target translation. motivated by work showing llms are useful for describing differences between sets of text @xcite , we construct one prompt per concept including all target-language variations and their respective lists of source-language sentences, and prompt each model to provide a natural language description of each target-language variation. 10 we generate rules from three instruction-tuned models: gemma-1.1 @xcite , llama-3 (ai at meta, 2024), and gpt-4 (openai, 2023); figure 1 provides an example rule set from gpt-4. for each language, we ask all three native speakers to label every gpt-4-generated rule for correctness. ## experiments we evaluate three instruction-tuned models, gpt-4, llama-3-8b-instruct, and gemma-1.1-7b-it; and two high-quality nmt models madlad-400-10b-mt @xcite and nllb-200-3.3b @xcite , sampling with temperature of 0. we hypothesize that models performing lexical selection will benefit from access to rules describing concept variations (section 3). thus, we evaluate instruction-tuned llms in 3 settings: (1) no rules provided, (2) with self-generated rules, and (3) with rules generated by gpt-4. for each setting, we instruct the model to explain its reasoning prior to selecting a lexical choice @xcite . 11 accuracy is measured as the proportion of sentences for which the model selects the correct lexical variation. we include a simple frequency baseline that always predicts the most common targetlanguage variation in our dataset for each concept during lexical selection. figure 2 shows results for each model. first, we find that sota nmt systems trained on our low-resource language pairs outperform all llms on telugu and hindi, while being comparable on afrikaans, galician, tamil, and farsi. for the remaining three languages, the best-performing llm, achieves a 4-15% absolute improvement in performance over the nmt systems. we find a large gap in performance between open-and closedweight llms, with the frequency baseline outperforming open-weight llms without rules in seven languages. llama-3-8b-instruct outperforms gemma-1.1-7b-it for all nine languages. self-generated lexical rules improve model performance on nearly all languages, with improvements being even more significant for the weaker open-weight models. this suggests that models can better reason over lexical choices if their parametric knowledge of concept variation is explicitly included as context in the prompt. while lexical selection with gpt-4 is significantly more expensive than with open-weight models, rule gen-eration has a one-time cost. when providing the open-weight models with rules acquired from the strongest model (gpt-4), we see total improvements up to 23% in accuracy, with these models performing close to or even exceeding gpt-4 on several languages. however, even when these highquality ( ## related work the most closely related work is by chaudhary et al. ( 2021 ), who study lexical selection for english to spanish and greek. they present a pipeline for collecting lexical selection data, train models to perform lexical selection, use a linear svm model to extract features as interpretable rules, and evaluate the efficacy of these rules in a second-language acquisition setting. in contrast, we use modern lms, generate natural language rules, and evaluate on several low-resource languages. we also curate a test set for lexical selection validated by native speakers of our target languages. 2023 ) evaluate llms on translations under polysemy, demonstrating that in-context learning and fine-tuning on ambiguous datasets improves translation. we study the potential for llms to resolve ambiguity arising from target-language concept variation, focusing on low-resource languages. prior work has shown improvements in llm translation quality by incorporating ground truth dictionary entries into prompts @xcite . we further demonstrate that models can accurately describe concept variations in lowresource languages using only parametric knowledge and example usages from source-language sentences. our experiments follow a line of work showing that modern llms exhibit non-english language capabilities, though these llms are often trained primarily on english data @xcite . ## conclusion we introduce dtails, a dataset of 1,377 sentence pairs with 9 language pairs that exhibit ambiguity in translation due to concept variation. using this dataset, we evaluate 3 popular instruction-tuned llms and 2 high-performing nmt systems on the task of lexical selection. out of nine languages tested, the strongest llm outperforms the nmt systems on three languages, has comparable performance on four languages, and fall short of these systems on two languages. no model is able to disambiguate the full set of sentences that native speakers can. we also present a simple approach to extract high-quality rules from language models, demonstrating improvements on lexical selection when lms are given access to rules. we find that providing weaker open-weight models with rules from a stronger llm can effectively bridge the gap to or even outperform the stronger model for several languages. future research could investigate additional applications of lexical rules in nmt and assess how these human-readable rules can assist l2 learners in vocabulary acquisition.