In-context Learning as Maintaining Coherency: A Study of On-the-fly Machine Translation Using Large Language Models (2023)

Abstract: The phenomena of in-context learning has typically been thought of as “learning from examples”. In this work which focuses on Machine Translation, we present a perspective of in-context learning as the desired generation task maintaining coherency with its context, i.e., the prompt examples. We first investigate randomly sampled prompts across 4 domains, and find that translation performance improves when shown in-domain prompts. Next, we investigate coherency for the in-domain setting, which uses prompt examples from a moving window. We study this with respect to other factors that have previously been identified in the literature such as length, surface similarity and sentence embedding similarity. Our results across 3 models (GPTNeo2.7B, Bloom3B, XGLM2.9B), and three translation directions (en \rightarrow {pt, de, fr}) suggest that the long-term coherency of the prompts and the test sentence is a good indicator of downstream translation performance. In doing so, we demonstrate the efficacy of in-context Machine Translation for on-the-fly adaptation.

## introduction the in-context learning paradigm describes a phenomena where large autoregressive language models perform a task when shown examples (known as prompts) in the prefix @xcite . in-context machine translation is a relatively new paradigm that uses large autoregressive language models to carry out the task of machine translation (mt) by being shown translation pairs in the prefix. from a practitioner's viewpoint, in-context learning presents itself as an attractive approach for rapidly adapting a translation model on-thefly. previous strategies for adapting a pre-trained mt model still require additional engineering or training of the model, e.g fine-tuning with in-domain data using adaptor layers @xcite . instead, simply changing the inputs to the model might be an effective way to adapt on-the-fly without any model modification. previous work assumes that the role of the prompt context is to allow the model to "learn by examples". this has led to formulating the task of prompt selection as selecting examples that are similar to the source sentence being translated. semantic similarity based on sentence embeddings @xcite and bm25 have been proposed to select examples to present as "demonstrations" @xcite . this approach was further expanded by @xcite who use a heuristic version optimizing for word coverage. translate english to french. english: a discomfort which lasts .. french: un malaise qui dure english: html is a language for formatting french: html est un langage de formatage ... ... english: after you become comfortable with formatting .. french: we focus on machine translation as a complex conditional generation task and offer an alternate perspective: the in-context paradigm depends on maintaining coherency. coherence is an aspect of natural language that reflects the overall semantic and syntatic consistency in a body of text @xcite . we investigate this by first exploring the model's behavior when showing matching and mismatching domains in the context and the test sentence. next we consider a stricter notion of coherency using a moving window of previous gold translations directly preceding the test source sentence to be next translated. our experiments compare the coherence factor with similarity based factors for prompt selection, additionally controlling for length @xcite which is typically overlooked but is important to consider for performance and available labeling (translation) budget. the contributions of this work are • we identify coherency of prompt examples with respect to test sentence as a critical factor for translation performance. experiments across 3 models (gptneo2.7b, bloom3b, xglm2.9b) and 4 domains (medical, social media, wikipedia, and ted talks) suggest that models perform better when prompts are randomly drawn from the same domain. ## factors which affect in-context mt we outline several factors studied in this paper related to example selection for in-context mt in figure 1 . while we emphasise the notion of coherence (section 2.2), by studying the domain factor (section 3.4) and local coherence (section 3.5), our experiments seek to compare this against other factors that have been highlighted in previous literature. namely, length (section 3.1), surface similarity (section 3.2) and semantic similarity (section 3.3). to demonstrate, in figure 1 : factors identified and studied in this paper. each domain has different length distributions (section 5.2). surface similarity and embedding similarity are associated ( ## further analysis and discussion in this section, we focus on gptneo2.7b and in the @xmath0. ## conclusion in-context learning has typically been thought of as learning from examples. in this work, we introduce a different perspective of coherency of the context with the test sentence. we found that 2 out of 3 models are able to adapt to different writing styles when the prompt bank and test set are matching/consistent in domain. experiments across 3 models and 3 languages show that a moving window is up to 2.6 bleu points better than previously reported similarity based retrieval methods from outside the document. from this perspective, the problem of prompt selection for in-context mt is one of maintaining a coherency for text generation. preliminary analysis on local coherence effects, and the presence of negative interference compared to the zero-shot setting, suggests avenues for future work on investigating more careful mechanisms for controlling in-context machine translation.