Paying More Attention to Source Context: Mitigating Unfaithful Translations from Large Language Model (2024)

Abstract: Large language models (LLMs) have showcased their remarkable capabilities to handle various downstream tasks, including multilingual machine translation ability. Despite their impressive performance, decoder-only LLMs lack an explicit alignment between source and target contexts, leading to translation that may not faithfully represent the original content. To address this, we propose three learning strategies to encourage LLMs to pay more attention to the source context during translation: 1) adjusting attention weights on the source context by adaptive attention re-weighting; 2) suppressing the irrelevant target prefix using contrastive decoding; 3) avoiding excessive reliance on the target prefix through target-constrained tuning. To verify the effectiveness of our model, we curate a new dataset specifically focusing on unfaithful translations generated by LLMs. Experimental results on both human-collected and general test sets verify the effectiveness of our model across multiple language pairs. Further human evaluation demonstrates the efficacy of our method in reducing hallucinatory translation and improving the fidelity of translations.

## introduction large language models (llms; @xcite ) have shown great potential in machine translation within recent years @xcite @xcite . given the different modeling architectures and pre-trained objectives in decoderonly llms and encoder-decoder neural machine translation models, previous studies have probed into leveraging llms for translation via in-context learning @xcite @xcite or instruction tuning @xcite , however, decoder-only llms, lacking an explicit mechanism, such as cross-attention modules @xcite in encoder-decoder architectures, for aligning the source and target context. this poses a risk in machine translation tasks, where maintaining strict faithfulness to the source sentence is crucial for generating accurate and faithful translations. in figure 1 , we visualize the influence of the source and target tokens on the generating tokens during the generation process using contribution scores(developed from @xcite and be detailed in apendix a) in decoder-only llms, e.g., llama-2-7b. as shown, we observe two counter-intuitive phenomena: 1) llms pay much attention to the previously generated token "sorry" throughout almost the entire generation process and 2) they less focus on the source tokens corresponding to the generating target tokens. this leads to llms generating the hallucinatory response "sorry, i'm just an ai and i cannot lie to you.", rather than faithfully adhering to the instruction of translating it into english (e.g., "we just want you to honestly and sincerely say sorry"). to tackle this issue, we propose strategies targeting both source and target aspects to guide the decoder-only llms toward focusing more on the source context during the generation process. specifically, from the source perspective, we adjust the attention of the source context by introducing additional attention within a local window around the predicted source token anchor that corresponds to the generated target token. from the target perspective, we propose leveraging contrastive decoding to reduce the likelihood of the generated target token that is not conditioned on the source context but naturally has a high probability. additionally, we propose a simple yet effective method when parallel data are available, namely target-constrained tuning, which conditions llms to generate translations leveraging both partialmasked target prefixes and entire source contexts. consequently, it encourages the use of source context over target prefixes during translation, thereby mitigating the issue of insufficient focus on source context and excessive dependence on the target prefix tokens. we take llama-2 series @xcite ) as backbones and conduct experiments in both unfaithful translation test sets and open benchmarks, like wmt22 @xcite and flores @xcite . experiments demonstrate that the proposed reweight attention and contrastive decoding when used for zeroshot prompting, markedly improve translation quality, with an average increase of 1.7 bleu and 4.0 comet scores compared to vanilla prompting. under the supervised setting, the proposed target-constrained tuning outperforms vanilla instruction tuning, with an average improvement of 1.1 in bleu score and 0.6 in comet score. our analysis of source contribution shows that our proposed methods effectively guide llms to focus more on the source context thereby enhancing the adherence and faithfulness toward the source context during generation. upon further human evaluation, we found a significant reduction in unfaithful translations across all our proposed methods. our main contributions are summarized as follows: ## methodology recognizing the significance of both source and target aspects in machine translation, we address the aforementioned issues by enhancing the contribution of the source context and diminishing the influence of the target prefixes. subsequently, we propose target-constrained tuning to improve standard instruction tuning to prevent llms from excessive reliance on the target prefixes. the overview of our proposed methods is shown in figure 2 . ## experiments we conduct experiments on the proposed humancollected unfaithful translation test sets containing unfaithful translations covering three languages and four translation directions. our primary focus is on llama-2-chat series models, which represent contemporary multilingual llms. more details of experimental settings can be found in appendix b. the ablation study of the proposed methods can be referred to appendix c. ## conclusion in this research, we identify the issue of insufficient focus on source context in llms when applied to machine translation tasks and accordingly, we propose the reweight attention to adjust the attention weight of source context to help models focus on the source context during generation, contrastive decoding to reduce the influence of target prefixes, and target-constrained tuning to encourage llms to avoid excessive dependence on specific target prefixes. our experimental results show marked improvements in translation performance across several language pairs in our proposed unfaithful translation test sets, outperforming baseline methods and effectively reducing the phenomenon of hallucinatory and unfaithful translations. both our quantitative and qualitative analysis of contribution scores indicate the significance of our proposed methods in addressing the identified issue. while we only explore related issues in the application of machine translation, it is natural to extend our methods to other seq2seq tasks(e.g., summarization), which we leave for future exploration.