Towards Effective Disambiguation for Machine Translation with Large Language Models (2023)

Abstract: Resolving semantic ambiguity has long been recognised as a central challenge in the field of Machine Translation. Recent work on benchmarking translation performance on ambiguous sentences has exposed the limitations of conventional Neural Machine Translation (NMT) systems, which fail to handle many such cases. Large language models (LLMs) have emerged as a promising alternative, demonstrating comparable performance to traditional NMT models while introducing new paradigms for controlling the target outputs. In this paper, we study the capabilities of LLMs to translate “ambiguous sentences” - i.e. those containing highly polysemous words and/or rare word senses. We also propose two ways to improve their disambiguation capabilities, through a) in-context learning and b) fine-tuning on carefully curated ambiguous datasets. Experiments show that our methods can match or outperform state-of-the-art systems such as DeepL and NLLB in four out of five language directions. Our research provides valuable insights into effectively adapting LLMs to become better disambiguators during Machine Translation. We release our curated disambiguation corpora and resources at https://data.statmt.org/ambiguous-europarl.

## introduction while the field of nmt has advanced rapidly in recent times, the disambiguation and translation of ambiguous words still remain an open challenge. notably, @xcite created a benchmark named dibimt to study the behaviour of state-of-the-art (sota) nmt systems when translating sentences with ambiguous words. 1 they reported that even the best-performing commercial nmt systems yielded accurate translations only 1 https://nlp.uniroma1.it/dibimt/public/ leaderboard ## conclusion in this work, we studied the capabilities of llms to handle ambiguity during machine translation. we choose seven of the most widely used foundation and instruction-tuned llms and compare accuracy with sota commercial and open-source nmt systems on the dibimt translation benchmark. out of 5 language directions, we report scores comparable to the sota on two (en-ru, en-it) and set a new sota on two others (en-zh, en-es). we then present two techniques that significantly improve disambiguation accuracy: in-context learning with similar contexts, and fine-tuning on an ambiguous corpus. we end the paper with an evaluation of overall mt quality. we hope the methods and findings shared in this work could guide future researchers studying ambiguity in translation.