Word Alignment as Preference for Machine Translation (2024)

Abstract: The problem of hallucination and omission, a long-standing problem in machine translation (MT), is more pronounced when a large language model (LLM) is used in MT because an LLM itself is susceptible to these phenomena. In this work, we mitigate the problem in an LLM-based MT model by guiding it to better word alignment. We first study the correlation between word alignment and the phenomena of hallucination and omission in MT. Then we propose to utilize word alignment as preference to optimize the LLM-based MT model. The preference data are constructed by selecting chosen and rejected translations from multiple MT tools. Subsequently, direct preference optimization is used to optimize the LLM-based model towards the preference signal. Given the absence of evaluators specifically designed for hallucination and omission in MT, we further propose selecting hard instances and utilizing GPT-4 to directly evaluate the performance of the models in mitigating these issues. We verify the rationality of these designed evaluation methods by experiments, followed by extensive results demonstrating the effectiveness of word alignment-based preference optimization to mitigate hallucination and omission. On the other hand, although it shows promise in mitigating hallucination and omission, the overall performance of MT in different language directions remains mixed, with slight increases in BLEU and decreases in COMET.

## introduction large language models (llms) have been evolving rapidly and showing predominant performance in many natural language processing (nlp) tasks @xcite @xcite . however, in machine translation (mt), the use of decoder-only llms is still limited due to issues such as model size @xcite and low-resource languages @xcite . conventional encoder-decoder mt models trained on parallel corpora still dominate in practice @xcite . one of the primary concerns of applying an llm to mt is reliability. although it does not happen frequently, an llm is known to hallucinate @xcite @xcite as it is pre-trained to predict the next token in very large-scale raw texts. specifically in mt, llmbased translation systems therefore could have the phenomena of hallucination and omission, which is also a long-term challenge in the field of mt @xcite , known as over-and under-translation. in particular, in the very recent wmt-2024 general machine translation task @xcite , a newly released llm-based mt model unbabel tower @xcite has achieved the highest accuracy in most language pairs, demonstrating the promise of llm in mt, but also showing the significance of the problem of hallucination and omission. as a result, we attempt to mitigate the hallucination and omission in llm-based mt to improve its practicality in this work. hallucination in mt occurs when information not present in the source text is generated in the translation, and omission occurs when some of the information in the source text is missed in the translation. as a related tool that explicitly aligns the source text and translation at the word level, word alignment is potentially positive for mt due to the nature of align and translate @xcite . the degree of coverage of the source text in translation could be a direct signal to identify the hallucination and omission in mt @xcite . figure 1 shows the normalized frequency of the coverage scores predicted by a word aligner. the examples that are annotated as "no hallucination or omission" tend to have a higher coverage score, while those in "full hallucination or omission" are more likely to have an extremely low coverage score. "small hallucination or omission" figure 1 : a preliminary experiment shows that higher coverage scores correlates to less hallucination and omission. the coverage scores are predicted by a word aligner @xcite . the human annotation of hallucination and omission is from halomi benchmark @xcite . details about the dataset and word alignment model can be found in §5.1. and "partial hallucination or omission" distribute in the middle. as the annotations are carefully made by humans and highly correlates to the coverage scores from the word aligner, this indicates that word alignment is a simple but promising direction to mitigate these phenomena. consequently, we propose word alignment preference (wap) that utilizes word alignment as a signal to optimize llm-based mt models. wap consists of three steps: diverse translation collection, preference data construction, and preference optimization. specifically, we collect diverse translations with multiple existing translation tools, select chosen and rejected examples with the word aligner @xcite , and optimize the model on preference data using direct preference optimization (dpo) @xcite . furthermore, the evaluation of hallucination and omission is challenging, and there is no existing evaluator specifically designed for this. improving the bleu and comet score does not necessarily mean reducing hallucination and omission because there are other factors such as mistranslation and fluency. in addition, hallucination is relatively infrequent, although very severe once it does occur. hence, to effectively evaluate it, we design extensive experiments that include testing on instances that potentially have the problem of hallucination and omission, and using gpt-4 as the evaluator with comprehensive analysis. experimental analysis demonstrates the effectiveness of wap in mitigating hallucination and omission in mt. in summary, the contributions of this work include the following: from the preliminary experiments in figure 1 we found that word alignment is a promising signal to mitigate it. preference tuning for llms. llms are capable of completing tasks in the zero-shot or few-shot manner @xcite . in addition, performance in downstream tasks can also be enhanced by fine-tuning them with instruction datasets @xcite @xcite . however, acquiring instruction datasets is costly, while obtaining preferences for llm responses is relatively easier @xcite . dpo @xcite directly optimize llm with preference data by removing an extra reward model. we utilize dpo in this work due to the ease of use and effectiveness. a contemporaneous preference-based method alma-r @xcite , introduces contrastive preference optimization to fine-tune llms specifically using reference-free mt metrics and human annotation as preference. alma-r focuses on improving general llm-based mt but we attempt to mitigate the hallucination and omission in mt. in addition, our preference data are made entirely automatically, which also draws the difference between alma-r and our work. the recently released llm-based unbabel tower @xcite has achieved the best performance in most language pairs in wmt-2024 @xcite , which may complement our findings in future work. word alignment. word-level information has been useful in many nlp tasks such as language pre-training @xcite , cross-lingual sentence embedding @xcite @xcite , finegrained visual language grounding @xcite @xcite , and particularly in word alignment for mt @xcite , which aligns the corresponding words in translations. word aligners based on pre-trained language models @xcite @xcite have outperformed previous ones based on statistical @xcite . wspalign @xcite ) is a pretrained word aligner that outperforms most of the previous ones; hence we use it in the experiments. ## conclusion the problem of hallucination and omission, a longstanding problem in mt, could become more severe when an llm is used because an llm itself could hallucinate or omit in nature. in this paper, our aim is to mitigate this problem in llmbased mt by optimizing the model toward a preference for better word alignment. we construct preference datasets by collecting translations using multiple mt tools and selecting the preference pair with a higher coverage score output by a word aligner. dpo is then utilized to optimize the model towards the word-aligned preference. as evaluation of hallucination and omission is challenging, we design experiments that include selecting hard instances and using gpt-4 to directly predict coverage score, ensuring an effective evaluation, which indicates that the proposed wap mitigates hallucination and omission, especially in hard instances. on the other hand, although wap shows promise in mitigating hallucination and omission, the overall performance of mt in different language directions remains mixed, with slight increases in bleu and decreases in comet.