In-context Examples Selection for Machine Translation (2023)

Abstract: Large-scale generative models show an impressive ability to perform a wide range of Natural Language Processing (NLP) tasks using in-context learning, where a few examples are used to describe a task to the model. For Machine Translation (MT), these examples are typically randomly sampled from the development dataset with a similar distribution as the evaluation set. However, it is unclear how the choice of these in context examples and their ordering impacts the output translation quality. In this work, we aim to understand the properties of good in-context examples for MT in both in-domain and out-of-domain settings. We show that the translation quality and the domain of the in-context examples matter and that 1-shot noisy unrelated examples can have a catastrophic impact on output quality. While concatenating multiple random examples reduces the effect of noise, a single good prompt optimized to maximize translation quality on the development dataset can elicit learned information from the pre-trained language model. Adding similar examples based on an n-gram overlap with the test source significantly and consistently improves the translation quality of the outputs, outperforming a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets.

## introduction in-context learning @xcite has recently received a lot of attention from the nlp research community due to its remarkable ability to utilize only a few input-output examples to perform many nlp tasks @xcite . for example, @xcite demonstrate that a 7.5b multilingual generative model, xglm, outperforms a supervised sequence-to-sequence baseline in 45 translation directions on the flores-101 machine translation benchmark @xcite using just 32 randomly sampled translation examples * work done during internship at meta ai research. as demonstrations. while these results are compelling, recent work has also shown that the performance and capability of a pre-trained language model (plm) can be highly sensitive to many factors, such as the choice of in-context examples @xcite , their ordering @xcite and the template @xcite . typically, in-context learning for mt uses examples that are randomly sampled from a small development set that resembles the domain of the test dataset. the effect of the aforementioned factors (such as the choice of the examples) on the translation quality of the plm hence remains unclear and unexplored. yet another crucial gap in using in-context learning for mt in the current literature is the effect of the domain of in-context examples on translation quality since out-of-domain generalization is a known and important challenge in mt @xcite . in this work, we systematically analyze how factors such as the choice and the number of few-shot in-context examples and their ordering impact mt output quality. we show that while noisy unrelated 1-shot example can have a significantly adverse effect on translation quality, a single prompt optimized to maximize the translation quality on a development set can sufficiently elicit task-based information from the plm. our analysis thus demonstrates the importance of selecting good examples for mt and raises the question: what are the properties of good in-context examples for mt? in that direction, our findings suggest that a well-formed meaning-equivalent translation example results in higher quality translation than randomly selected in-context examples. motivated by the use of translation memory in computer-aided translation @xcite and its usage in computational approaches to machine translation @xcite @xcite , inter alia), we retrieve similar examples to the test source from a datastore that includes pairs of the source text and their corresponding translations via bm25, an unsupervised efficient retriever to provide additional context to the model. we propose a novel incontext example selection and re-ranking strategy to maximize the coverage of the source n-grams in the retrieved examples. experiments on wmt'19 english↔german and english↔russian datasets show that our proposed strategy can consistently improve the translation quality over the outputs generated using bm25 retrieved examples. combining optimized 1-shot task-level with examplespecific in-context examples using a simple concatenation strategy further improves translation quality, outperforming state-of-the-art inferenceadapted nearest-neighbor mt models (knn-mt) on two out-of-domain datasets (medical and it) while being memory and compute efficient as our approach does not require constructing and querying a dense token-level datastore. ## background: in-context learning generating translations from large-scale multilingual language models like mgpt @xcite , xglm @xcite or alexatm 20b @xcite requires conditioning the decoder-only language model with in-context parallel examples. these examples serve two purposes: a) providing the model with the format and knowledge of the task (task-level) and b) guiding the output generation via providing useful information about the unseen source sentence (example-specific). this is different from the standard sequence-to-sequence models, where the task is always known, and the model learns generalizable patterns from the input-output examples to perform the task (in this case, translation) for the unseen source text. source: welche risiken sind mit poulvac flufend h5n3 rg verbunden? template: {source text} = {target text}. example-specific: welche risiken sind mit sebivo verbunden? = what are the risks associated with sebivo? task-level: bei promess1 werden drei hauptziele verfolgt. = promess1 has three main objectives. formally, given k in-context examples {x i , y i } k 1 the prefix input or the prompt, x p j , is generated by concatenating the demonstration examples {(x i , y i )} k 1 to the test input, x s j according to a template, p (see ## prompt selection ideally, good in-context examples can trigger the pre-trained language model to generate the desired output and also elicit the information learned during pre-training @xcite . @xcite show that, for classification tasks, the incontext examples provide information about the task (the distribution of the input text, the label space, and the format of the task) and that the model does not rely on these examples to generate the final output. however, their analysis is limited to a) classification tasks and 2) randomly sampled in-context examples. prior work has also shown that the order of these in-context examples can also lead to high variance in downstream performance @xcite . however, less is understood about how these factors impact text generation tasks like mt. do we need multiple incontext examples? what makes good in-context examples for mt? how sensitive is the model to the order of the prompts? in this work, we aim to better understand the impact of prompt selection on the translation quality of the outputs. given a training dataset consisting of n parallel examples @xmath0, y i } n @xmath1, and a test source x j , we select a subset of m informative samples to form a prompt which either provides task-level and/or example-specific information as discussed below. ## results ## analysis 6.1 task-level example selection choice of few-shot examples we show the distribution of output quality as measured by bleu when using 100 different examples as prompts in figure 2 . across all four language pairs, there is a large variation in bleu scores (up to 20 bleu), where noisy or unrelated prompts can lead to significantly worse output quality. given that most existing parallel corpora are web-crawled and the quality of bitext can vary significantly across different language pairs @xcite , randomly sampled examples can under-estimate the translation quality attainable by prompting the plm. properties of good task-level prompts our manual analysis on the best task-level prompts suggests that any well-formed and meaning-equivalent translation @xcite could make a good task-level prompt (see examples in appendix ## conclusion we investigate the choice of in-context examples selection for mt in both in-domain and out-ofdomain settings. we propose a novel recall-based re-ranking approach to utilize similar training examples as prompts and show their efficacy across multiple datasets and domains. our findings show that task-level prompts can provide a complementary advantage to example-specific prompts, outperforming a strong knn-mt baseline in 2 out of 4 out-of-domain datasets while being memory and compute efficient. our manual analysis of the generated outputs reveals that the plm can mimic the style of the in-context examples provided and can be used for template-based translation synthesis. these results allow future research to evaluate the potential of generating diverse and style-specific outputs for mt.