An Empirical Study of In-context Learning in LLM s for Machine Translation (2024)

Abstract: Recent interest has surged in employing Large Language Models (LLMs) for machine translation (MT) via in-context learning (ICL) (Vilar et al., 2023). Most prior studies primarily focus on optimizing translation quality, with limited attention to understanding the specific aspects of ICL that influence the said quality. To this end, we perform the first of its kind, exhaustive study of in-context learning for machine translation (MT). We first establish that ICL is primarily example-driven and not instruction-driven. Following this, we conduct an extensive exploration of various aspects of the examples to understand their influence on downstream performance. Our analysis includes factors such as quality and quantity of demonstrations, spatial proximity, and source versus target originality. Further, we also investigate challenging scenarios involving indirectness and misalignment of examples to understand the limits of ICL. While we establish the significance of the quality of the target distribution over the source distribution of demonstrations, we further observe that perturbations sometimes act as regularizers, resulting in performance improvements. Surprisingly, ICL does not necessitate examples from the same task, and a related task with the same target distribution proves sufficient. We hope that our study acts as a guiding resource for considerations in utilizing ICL for MT. Our code is available on https://github.com/PranjalChitale/in-context-mt-analysis.

## introduction large language models leverage in-context learning to effectively solve diverse downstream tasks @xcite @xcite @xcite . this inference-only method involves conditioning the model with task-specific demonstrations consisting of input-output pairs within the prompt before the actual test example. most recently, mt via icl has become popular @xcite , however, there is still a limited understanding of how icl works and its aspects in this context. @xcite make some initial explorations on partial test sets, but there are still open questions pertaining to whether icl is example-driven or instruction-driven, whether all examples contribute equally or not, and its potential to enforce control over the generation to regulate the pre-training biases observed in models. to answer these questions, in this paper, we conduct experiments by perturbing either the instructions or the demonstrations while keeping the other clean to assess the model's ability to perform tasks appropriately. this allows us to understand if clear instructions can guide the model effectively even when examples are perturbed and, conversely, if clean examples can steer the model to perform the appropriate task despite the instruction being misleading. we assess 6 models across 2 language families, spanning 12 language pairs on complete test sets across various noise thresholds by considering realistic perturbation attacks. when perturbing examples, we study 1) whether icl treats all examples equally or favors those in spatial proximity to the test example, 2) whether utilizing in-context examples from related tasks can effectively guide models in performing another task to simulate the unavailability of demonstrations for some language pairs, 3) whether various choices of auxiliary source language for the demonstrations influence downstream mt, 4) whether models can implicitly make associations based on contextual information via a transitive mt setup and finally, 5) whether smaller models are susceptible to contextual misinformation or if their pre-training biases act as safeguards against it. our work aims to holistically understand icl and its driving factors for mt tasks with experiments spanning over 25k runs across various models and language directions. we observe that examples significantly influence icl, while instructions have a limited impact. notably, target distribution plays a more significant role than source distribution in icl demonstrations. while perturbations might seem potentially harmful, in some cases, they can act as a form of regularization, particularly evident in the llama 2 7b model. spatial proximity emerges as a critical factor, implying that clean examples should be placed closer and noisy ones farther during in-context example selection for optimal downstream performance. furthermore, our findings suggest that examples from related tasks can suffice for icl in mt, with the constraint that the target language of the demonstrations matches the language of the test example while the choice of the source language is inconsequential. the directionality of the demonstrations has minimal impact, with both target-original and source-original demonstrations being equally effective. icl has the potential to override semantic priors and may be exploited to misguide models or generate misinformation. we believe our study offers valuable insights for practitioners and would be widely generalizable. ## related works in-context learning: large language models have demonstrated strong performance in various downstream tasks through supervised fine-tuning on task-specific data @xcite @xcite . icl @xcite @xcite @xcite is a training-free method that has emerged as a cost-effective approach given the increasing size of llms (≥ 7b parameters). icl aims to implicitly learn the task-solving abilities through a few hand-crafted demonstrations. these emergent capabilities @xcite are driven by the distribution of the pretraining data @xcite . intuitively, icl can be considered analogous to performing a gradient descent implicitly during the inference @xcite @xcite . @xcite explored few-shot prompting strategies for mt task on palm @xcite suggesting that the downstream mt performance is largely correlated with the quality of demonstrations. subsequently, several recent works have also explored icl capabilities of llms for the mt task @xcite @xcite . however, none of these works delve into a fine-grained examination of the underlying factors influencing the model performance; a gap that we fill with our study. factors impacting in-context learning: several aspects of the demonstrations, such as input distribution, output distribution, or input-output alignment, can play a significant role in the icl performance of llms. while min et al. (2022) observed limited significance of label correctness in demonstrations for classification tasks, however, yoo et al. (2022); @xcite argued that icl is sensitive to input-label mapping and can affect downstream performance. furthermore, @xcite showed that icl can override semantic priors, being influenced even by semantically unrelated input-label mappings, particularly in larger models. while prior works @xcite @xcite have explored various aspects of icl for nlu tasks, it is crucial to examine its applicability to nlg tasks. @xcite , which is most related to our work, found that syntactic perturbations to demonstrations affect mt performance, with target text distribution having more impact than source text distribution. however, their experiments were limited to gpt-3 @xcite with only 100 test examples per direction. our study substantially expands on this by extensively experimenting across the multiple axes mentioned earlier. perturbation and robustness: text perturbation attacks @xcite @xcite @xcite are frequently used to assess the resilience of models to input alterations, distinguishing them from adversarial attacks @xcite @xcite with their intent to simulate realworld noise rather than to explicitly deceive models. these attacks, applied at either word level @xcite or character level @xcite @xcite , assess nmt model robustness through strategies like introducing random or linguistically aware perturbations, such as frequently confused characters. additionally, perturbations may also involve random removal or addition of punctuations @xcite . closest to our work is the work by moradi and samwald (2021), however, our approach differs from theirs as we only perturb demonstrations while leaving test examples as is. ## methodology our experimentation first focuses on understanding whether in-context learning for mt is instructiondriven or example-driven, and then expands on the latter, exploring various aspects of examples. we categorize our experimentation along two principal axes: instruction perturbations and demonstration perturbations. ## experimental setup this section describes the different pre-trained models, evaluation benchmarks, in-context prompts, and decoding hyperparameters used for our current study. ## results and analysis we now describe results that attempt to reveal the various factors that influence icl for mt. ## conclusion although icl has demonstrated good performance and is widely employed across different tasks, there is limited understanding of the aspects that influence the downstream performance of icl. in this work, we aim to bridge this gap for mt by investigating the central question of whether icl for mt is example-driven or instruction-driven and subsequently examining the role of different aspects of icl. we find that: (1) icl is primarily example-driven and choice of instruction has limited impact, with target distribution of the demonstrations being the most influential (2) perturbation to demonstrations is generally detrimental but can have a regularization effect in some cases (3) spatial proximity to the test example is an important factor in icl (4) demonstrations having the same target language can serve as a reliable proxy and choice of the source language of the demonstration is inconsequential (5) the directionality of the demonstrations has minimal impact. (6) icl can potentially be exploited to mislead even smaller models, highlighting the need for further research to make models robust in-context learners.