Contrastive Decoding Reduces Hallucinations in Large Multilingual Machine Translation Models (2024)

Abstract: In Neural Machine Translation (NMT), models will sometimes generate repetitive or fluent output that is not grounded in the source sentence. This phenomenon is known as hallucination and is a problem even in large-scale multilingual translation models. We propose to use Contrastive Decoding, an algorithm developed to improve generation from unconditional language models, to mitigate hallucinations in NMT. Specifically, we maximise the log-likelihood difference between a model and the same model with reduced contribution from the encoder outputs. Additionally, we propose an alternative implementation of Contrastive Decoding that dynamically weights the difference based on the maximum probability in the output distribution to reduce the effect of CD when the model is confident of its prediction. We evaluate our methods using the Small (418M) and Medium (1.2B) M2M models across 21 low and medium-resource language pairs. Our results show a 14.6 ± 0.5 and 11.0 ± 0.6 maximal increase in the mean COMET scores for the Small and Medium models on those sentences for which the M2M models initially generate a hallucination., respectively.

## introduction hallucinations are a rare but problematic phenomenon in nmt (neural machine translation) whereby the target side output is repetitive or fluent but not grounded in the source sentence @xcite . even though hallucinations are rare in nmt, they are a significant problem as they undermine trust in deployed nmt systems. hallucinations occur when the target side sentence is detached from the source side sentence @xcite @xcite , or in other words, when there is a low contribution of the source sentence to the generation of the target sentence. previous work on mitigating hallucinations has focused on sampling translations and reranking them according to quality metrics @xcite . separate to this, @xcite proposed contrastive decoding (cd) as a way of mitigating bad behaviour (such as excessive repetition and low diversity) when generating from unconditional language models. cd is a decoding algorithm that maximises the difference between the log probabilities of a strong expert and a weak amateur model (equivalent to maximising the ratio of probabilities). a threshold is applied so that decoding follows the expert when it is more confident. the intuition behind cd is that the amateur model is more prone to certain types of low-quality generation, so by subtracting the log probabilities, these are removed. we hypothesise that by using cd with an amateur, which is prone to source detachment, we can mitigate hallucinations in nmt. in order to create an amateur with low source attachment, we experiment with different strategies for reducing the role of cross-attention. the simplest is the no encoder strategy, where the amateur is a decoder-only version of the expert. in our other strategies, we retain the encoder and crossattention but impose uniform attention, remove attention from the most highly attended source position, or scale down all cross-attention values. in contrast to unconditional generation, nmt should be more strictly grounded in the source sentence. additionally, hallucinations only account for a small proportion of translations, and hence, mitigation of hallucinations must not come at the cost of reduced performance on other sentences. as such, increasing the diversity of the translations is less desirable than it is in unconditional generation. ideally, cd would only take effect when a model is hallucinating. to address this issue, we experiment with a novel variant of cd that dynam-ically adjusts the subtraction's magnitude based on a distribution's maximum value. we evaluate our approach on large multilingual models, which have recently been shown to be prone to hallucinations @xcite . specifically, we use the m2m family of models @xcite and consider the 418m (small) and 1. @xcite versions. we summarise our contributions as follows: hallucination detection we discuss hallucination detection as it relates to our experimental setup. guerreiro et al. (2023b); dale et al. (2023); raunak et al. (2021) all evaluate different methodologies for automatically identifying hallucinations and demonstrate the effectiveness of alti+ as a hallucination detection method. hallucination mitigation guerreiro et al. 2023a propose using a different fallback model when a hallucination is detected. other methods rely on sampling and re-ranking translations, for example, using comet @xcite to mitigate hallucinations @xcite . compared to our methodology, this approach relies on an additional outside model to rank sentences to achieve the best performance. additionally, both works only evaluate on a small @xmath0, whereas we evaluate on large-scale multilingual models. in contemporaneous work, sennrich et al. (2023) uses a similar cd approach to mitigate hallucinations. hallucinations are evaluated by counting the proportion of segments with @xmath1. unlike our work, the authors use the same model as the amateur but supply it with randomly selected inputs. in contrast, we use different models as the amateur, supplied with the same inputs. randomly selecting another source segment is potentially less stable than using a model as an amateur, as individual translations can depend on the selection of the source segment. this work also compares different amateurs and techniques for combining the expert and amateur distributions, whereas @xcite places additional focus on off-target translations. our work can thus be seen as complementary to theirs. ## methodology we first describe cd as proposed by @xcite , then discuss our proposed improvements (normalisation and dynamic weighting), and finally motivate the amateur models that we use in our experiments. ## results first, we present results comparing our different experimental setups by combining all language pairings. we compare amateur models using the original cd approach before reporting on the effects of our additions to the cd algorithm. next, we compare the performance across languages by looking at the distributions of comet scores and presenting qualitative examples. finally, we use the hallucination detection suite to report the number of detected hallucinations when using cd and briefly compare our work to the contemporaneous work of @xcite . for completeness, we also report spbleu, chrf++ and comet for our selected approach in appendix b. ## conclusions this paper applies cd decoding to the task of mitigating hallucinations in nmt. we show that cd improves the mean comet scores of sentences for which the m2m translation models generate hallucinations. our results also support our hypothesis that a key part of effectively using cd to mitigate hallucinations is restricting decoder access to the encoder outputs in amateur models, simulating target detachment from the source. additionally, we experiment with decreasing the adverse effect of cd on sentences for which the m2m models already generate good translations by dynamically changing the weight hyperparameter, which scales the subtraction of the amateur probabilities. we show that dynamic weighting decreases the changes to translations generated compared to the expert, but this comes at the cost of repairing fewer hallucinations. improvements to the dynamic approach would require a model-based metric that identifies hallucinations at the token level. as such, we recommend the original approach and either removing the encoder outputs or scaling down the cross-attention weights. our brief comparison with source-contrastive decoding suggests that it maybe a better candidate, assuming the stochastic nature of the decoding process is acceptable for the application. in light of the adverse effects on non-hallucinations, we also suggest using cd only when a hallucination is detected, for example, with a hallucination detection pipeline. any 'fixed' hallucinations should also be flagged if cd is used in a deployed system. end users should be made aware that the translation was originally a hallucination and may still contain translation errors.