Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding (2024)

Abstract: Hallucinations and off-target translation remain unsolved problems in MT, especially for low-resource languages and massively multilingual models. In this paper, we introduce two related methods to mitigate these failure cases with a modified decoding objective, without either requiring retraining or external models. In source-contrastive decoding, we search for a translation that is probable given the correct input, but improbable given a random input segment. In language-contrastive decoding, we search for a translation that is probable, but improbable given the wrong language indicator token. Experiments on the massively multilingual models M2M-100 (418M) and SMaLL-100 show that these methods suppress hallucinations and off-target translations, reducing the number of translations with segment-level chrF2 below 10 by 67-83% on average across 57 tested translation directions. In a proof of concept on out-of-English translation, we also show that we can suppress off-target translations with large language models. We release code upon acceptance.

## introduction hallucinations are a long-standing well-known problem in machine translation (mt) @xcite and natural language generation @xcite . while there has been extensive research on their identification and mitigation @xcite @xcite @xcite , they still persist as an issue, especially in low-resource settings. contrastive conditioning has previously been used for analysing specific translation errors such as disambiguation errors and undertranslation @xcite . the main translate ʻi like applesʼ to french ## method to suppress hallucinations, we pair each input x with a randomly selected input segment x ′ . 2 rather than finding a translation that maximizes p(y |x), we search for one that both maximizes p(y |x) and minimizes p(y |x ′ ). we add a hyperparameter λ to control the strength of this contrastive penalty, yielding eq. 1. we denote this source-contrastive decoding. off-target translations are a common failure mode in multilingual mt systems @xcite . they have been linked to the predominance of english in the training of multilingual systems @xcite . production of text in the source language, often a copy of the input, is connected to the occurrence of copying in the training data, and the high probability of continuing to copy once a copy has been started @xcite . the majority of multilingual mt systems use special tokens to indicate the target language, following @xcite . 3 to penalize output in the wrong language, we can add contrastive inputs that only vary the language indicator token. let l y be the target language. we replace its indicator token with contrastive variants l y ′ ∈ l c for languages we wish to suppress. based on the predominant off-target languages in multilingual mt @xcite , our set of contrastive languages l c consists of english foot_3 and the respective source language. this results in eq. 2. we refer to decoding with contrastive translation directions as language-contrastive decoding. we can combine source-contrastive and languagecontrastive decoding by summing all contrastive variants, and refer to the weights as λ src and λ lang . ## ablation studies the fact that we pick contrastive inputs from the test sets at random raises a few questions about this approximation. we repeated the translation with m2m-100 across all 57 translation directions 3 times and find that the standard deviation is minimal (0.0107 for chrf2). using a single random input as a contrastive variant is a heavy approximation, but our ablation study in ## application to large language models in this section, we demonstrate that our method can be applied to large language models (llm). previous work has achieved competitive translation quality for some directions by prompting models such as palm @xcite , gpt @xcite or bloom @xcite . however, llm-based translation is still prone to hallucination and off-target translation @xcite . our demonstration is based on the llama 2 model family @xcite and specifically the instruction-tuned version (llama chat), exploiting the fact that mt examples were among the data used for instruction tuning @xcite . we generate translations by instructing the model to translate a segment into a given language, force-decoding the line "sure, here's the translation:", and then decoding until 13 we divide λsrc by the number of contrastive inputs. the next line break. the prompt we used is detailed in appendix e. when using this simple prompting approach in the en-de direction, we find that off-target output in english is very common. moreover, providing a 1-shot example in the prompt, while improving translation quality, does not prevent the off-target issue. we thus apply language-contrastive decoding and add a contrastive prompt that instructs the model to "translate" into english instead of german. the decoding objective is analogous to eq. 2. we use 4-bit precision @xcite and greedy decoding. figure 2 shows the percentage of off-target output for different λ lang . generally, we observe that the off-target rate falls with increasing λ lang , demonstrating the effectiveness of our method with llm prompting. english-french (appendix c) has similar results. in terms of overall translation quality, we find that language-contrastive decoding improves chrf2 and spbleu and only becomes detrimental for λ @xmath0.7 (appendix d). hallucination mitigation is more difficult, especially if we assume that models are already trained with best practices, and focus on training-free methods. several studies use external models for mitigation, e.g. using other translation models as a fallback @xcite , or sample reranking based on quality estimation (qe) models @xcite . our method has the advantage of not requiring external models, and we note that modern qe metrics are themselves prone to score certain hallucinations highly @xcite . mitigation methods that do not rely on external models are typically sampling-based. @xcite report that even the translation model's own sequence probability can be used for sample reranking. a consensus translation can be identified via sampling-based minimum bayes risk decoding (eikema and aziz, 2020), which benefits from the fact that hallucinations are dissimilar from each other @xcite . ## conclusion this paper shows that certain failure modes of mt can be addressed by contrastive decoding objectives that use pairs or sets of inputs for the prediction. specific contrastive inputs address specific errors, and we introduce strategies to mitigate hallucinations and off-target translation. future work could expand on our work by exploring if other mt failure modes can be mitigated with appropriate contrastive inputs, or if other forms of control can be improved. for example, for models that use domain indicator tokens @xcite , we could perform domain-contrastive decoding and achieve stronger domain control. beyond mt, we expect that source-contrastive decoding can also be useful for other tasks, e.g. to penalize over-generic responses in dialogue systems.