Analyzing Context Contributions in LLM -based Machine Translation (2024)

Abstract: Large language models (LLMs) have achieved state-of-the-art performance in machine translation (MT) and demonstrated the ability to leverage in-context learning through few-shot examples. However, the mechanisms by which LLMs use different parts of the input context remain largely unexplored. In this work, we provide a comprehensive analysis of context utilization in MT, studying how LLMs use various context parts, such as few-shot examples and the source text, when generating translations. We highlight several key findings: (1) the source part of few-shot examples appears to contribute more than its corresponding targets, irrespective of translation direction; (2) finetuning LLMs with parallel data alters the contribution patterns of different context parts; and (3) there is a positional bias where earlier few-shot examples have higher contributions to the translated sequence. Finally, we demonstrate that inspecting anomalous context contributions can potentially uncover pathological translations, such as hallucinations. Our findings shed light on the internal workings of LLM-based MT which go beyond those known for standard encoder-decoder MT models.

## introduction large language models (llms) have reached stateof-the-art performance in machine translation (mt) and are making significant strides toward becoming the de facto solution for neural mt @xcite . compared to the classical standard approach using encoder-decoder models @xcite , llms are typically decoder-only models parameterized by billions of parameters. remarkably, llms have demonstrated the ability to perform translation tasks without being explicitly trained for them, instead leveraging in-context learning (icl) through demonstrations of the task @xcite @xcite @xcite . yet, there is a gap in the literature on understanding the internal workings of llm-based mt. previous interpretability research on mt has been limited to traditional, specialized encoder-decoder models @xcite @xcite @xcite , and while substantial work has investigated icl in other tasks, such as classification @xcite @xcite and question answering @xcite @xcite , the mechanisms by which llms leverage parts of context in mt remain largely unexplored. in this work, we aim to fill this research gap by contributing towards a better understanding of how llms utilize different parts of the provided context (e.g., few-shot examples, the source text, or previously generated target tokens) in mt. while previous work conducted on understanding the impact of context in mt largely focuses on performing modifications on the llm input and measuring performance drop @xcite , we take instead an attribution-based approach @xcite , tracking the input tokens' relevance in all parts of the context-this allows us to estimate how different parts of context contribute to the generated translations, providing a more fine-grained analysis of context utilization. we study several key aspects of context utilization in mt using general purpose llama-2 models @xcite and tower models @xcite -a suite of models specifically adapted for translation tasks. first, we investigate how different input parts contribute to the translated sequence. next, we explore whether the provided few-shot examples contribute equally to the translated sequence. we also analyze if undergoing adaptation via continuous pretraining @xcite @xcite on relevant multilingual and parallel ## problem formulation in this section, we introduce icl and describe how we employ the alti method @xcite to measure the contribution of each input part in the context to the translated sequence. ## experimental setup we provide an overview of the models and datasets used throughout our study, as well as important considerations on how we prompt the models. ## how do different context parts contribute to the translated sequence? in this section, we conduct a top-level analysis by measuring and comparing the contributions of different input parts to the generated translation. ## examining positional bias over the provided few-shot examples motivated by the findings from the previous section, we now closely inspect properties of the positional bias in few-shot example contributions. ## how do context contributions evolve during the generation process? in the previous sections, we examined which parts of the provided context have the greatest influence on the translated sequence. we now shift our focus to explore how these context contributions evolve across different stages of the generation process. ## conclusion we have comprehensively studied context contributions in llm-based mt using the general purpose llama-2 and translation-specialized tower models, exploring a broad range of key aspects, including investigating how different parts of context contribute to generated translations, and how these contributions evolve during the generation process. our findings reveal a strong positional bias, where earlier few-shot examples in the context have higher contributions to the translated sequence, both at the sentence level and across different generation stages. interestingly, our experiments show that this bias is shrunk by continuous pretraining on task-specific data. moreover, we reveal that the source part of each few-shot example has higher contribution compared to its corresponding target, irrespective of the translation direction. finally, we stress the importance of source-part contributions by demonstrating that anomalous contributions can potentially uncover pathological translations, such as hallucinations. we believe our work not only provides insights into the internal workings of llm-based mt, but also draws important connections to classical encoder-decoder nmt models. to support future research on this topic, we are open-sourcing our code and releasing all data used in our analysis.