Dissecting In-Context Learning of Translations in GPT -3 (2023)

Abstract: Most of the recent work in leveraging Large Language Models (LLMs) such as GPT-3 for Machine Translation (MT) has focused on selecting the few-shot samples for prompting. In this work, we try to better understand the role of demonstration attributes for the in-context learning of translations through perturbations of high-quality, in-domain demonstrations. We find that asymmetric perturbation of the source-target mappings yield vastly different results. We show that the perturbation of the source side has surprisingly little impact, while target perturbation can drastically reduce translation quality, suggesting that it is the output text distribution that provides the most important learning signal during in-context learning of translations. We propose a method named Zero-Shot-Context to add this signal automatically in Zero-Shot prompting. We demonstrate that it improves upon the zero-shot translation performance of GPT-3, even making it competitive with few-shot prompted translations.

## introduction recent work has put into question the importance of the correctness of demonstrations for prompting in large language models (llms) @xcite . one key conjecture is that the latent zeroshot capabilities of llms might be considerably higher than their observed zero-shot capabilities for a range of tasks @xcite . one way to elicit higher zero-shot performance is to qualify the role of demonstration attributes towards task performance and then simulate such in-context learning signals in a zero-shot manner. however, realizing this goal hinges on explicitly dissecting the role of various demonstration attributes (format, inputs, outputs, input-output mapping) towards task performance within fewshot in-context learning. in this work, we explore these questions for the task of machine translation (mt). our line of inquiry is orthogonal to finding the most useful samples for few shot learning, a topic that has received considerable attention for eliciting better translations from llms @xcite . our contributions are: our work is related to two key themes, namely prompting llms for translation and analysis of in-context learning in llms. in this section, we situate our work within these two themes. llm prompting for mt: llms have achieved close to the state-of-the-art translation performance under few-shot prompting @xcite . most of the work for prompting in mt has focused on selecting the training or development instances to be used as examples during prompting. @xcite experiment on palm @xcite and find that quality of examples is the most important factor in few-shot prompting performance. high-quality, in-domain examples to better qualify the role of certain demonstration attributes for in-context learning of translations. analyzing in-context learning: theoretical and empirical investigation of in-context learning is an ongoing research endeavor @xcite @xcite . @xcite demonstrate that label correctness in demonstrations is of limited importance for open-set classification tasks, while @xcite show that negated labels do matter. our experiments differ from these works both on the choice of the task (translation, which has an exponential output space) as well as on the types of perturbations applied to the demonstrations. ## the role of demonstration attributes to produce outputs for a specific task, llms are typically prompted with demonstrations (inputoutput examples pertaining to the specific task) appended with the test input. similar to @xcite , we posit that there exist four aspects of demonstrations of the translation task that provide a learning signal: the input-output mapping, the input text distribution, the output text distribution and the format. in this section, we conduct an empirical investigation on how llms such as gpt-3 leverage the demonstrations provided to them for the task of translation by perturbing the inputoutput (source-target) mappings provided during prompting. through these experiments, we hope to compare the importance of three key demonstration attributes -the input text distribution, the output text distribution and their mapping for translation. ## zero-shot-context for translation previously, we demonstrated that the most important demonstration attribute for in-context learning of translations is the output text distribution. in this section, we present a method of providing this learning signal in a zero-shot manner. our experiment here represents an inverse of experiments in section 3, i.e., here we add a useful learning signal to zero-shot prompting, rather removing learning signals from few-shot prompting to gauge their importance. we present a method named 'zero-shot-context' and show that it greatly improves upon zero-shot performance for gpt-3, eliciting performance competitive even with few-shot prompting. note that this method is one example of adding a particular signal in zero-shot prompting and that there could be multiple ways to add such a signal to bolster zero-shot performance including direct instruction finetuning on the translation task. however, we leave a thorough analysis of improving zero-shot translation performance by adding relevant signals from demonstrations to future work and focus only exploring on our key hypothesis. ## summary and conclusions we analyzed the relative importance of demonstration attributes as learning signals within few-shot in-context learning of translations in llms from the gpt family. we demonstrated that the critical learning signal arrives from the output text distribution, followed by the input-output mapping, while the input text distribution matters for little. we use this finding to propose zero-shot-context, a method that tries to automatically generate the critical learning signal. zero-shot-context greatly improves upon zero-shot translation quality in gpt-3, further validating our findings. we hope that our work could serve as a useful contribution towards better understanding of in-context learning of translations in llms.