CTQS corer: Combining Multiple Features for In-context Example Selection for Machine Translation (2023)

Abstract: Large language models have demonstrated the capability to perform on machine translation when the input is prompted with a few examples (in-context learning). Translation quality depends on various features of the selected examples, such as their quality and relevance, but previous work has predominantly focused on individual features in isolation. In this paper, we propose a general framework for combining different features influencing example selection. We learn a regression model, CTQ Scorer (Contextual Translation Quality), that selects examples based on multiple features in order to maximize the translation quality. On multiple language pairs and language models, we show that CTQ Scorer helps significantly outperform random selection as well as strong single-factor baselines reported in the literature. We also see an improvement of over 2.5 COMET points on average with respect to a strong BM25 retrieval-based baseline.

## introduction large language models (llms) trained on massive amounts of textual data have demonstrated impressive performance on a wide range of nlp tasks despite not being explicitly trained on any of them @xcite @xcite @xcite . these capabilities of the model are elicited using in-context learning, where the model is prompted with task instructions and demonstration examples followed by the input. the task's output for the given input is simply the next sequence of tokens sampled from the language model. recently, in-context learning has also been explored for machine translation @xcite @xcite . many of these models have shown encouraging results for translation, particularly for highresource languages. this achievement is impressive given that the models have not been inten-tionally supplied with parallel data, and their training data predominantly consists of english content. however, performance on low-resource languages and translation out of english is an unresolved major challenge, where issues like hallucination and adequacy gaps have been observed. on the other hand, llm translations are more fluent and paraphrastic, and they handle long-distance reordering better -especially when translating into english @xcite . an important aspect of in-context learning is the creation of the prompt. the prompt typically consists of two parts: the prompt template that helps the model understand the task and the in-context examples that aid in the better translation of the input source sentence. the in-context examples can be selected for each input source from an example database/datastore that contains parallel sentence pairs. the number, order, and choice of examples can affect the translation quality @xcite . specifically, various features of examples like the translation quality of the sentence pair, the length of sentences, its semantic similarity to the input, etc. can contribute to the translation quality. previous approaches consider only individual features when selecting examples. however, such an approach could be sub-optimal as it ignores the relevance of other features to the translation quality. for instance, assume that we choose examples based on the quality of the translation pairs. the selected sentences may be short, potentially offering limited information for the translation of the input. moreover, the selection depends on the chosen translation quality metric. given that translation quality metrics are imperfect, a different metric might have led to a different selection of examples. a better approach would be to select examples based on a diverse set of features and different views of measuring the same feature (e.g. translation quality could be measured by various metrics such as labse @xcite or comet-qe @xcite ) to maximize translation quality. in this work, we explore example selection based on multiple features. our contributions are: 1. we propose selecting examples based on a scoring function that integrates evidence from different features. we propose ctq scorer, a regressionbased scoring function that estimates the translation quality given the example and the test input. 2. given the absence of manually annotated quality scores for training the proposed regression model, we propose a novel method for creating training data. we estimate contextual translation quality on a held-out set by 1-shot prompting on the llm using the input source from the held-out set with an example from the database as context. these (input source, example, quality score) tuples serve as training data for the regression model. 3. we show that combining evidence from multiple features selects examples that improve translation quality compared to the use of individual features. ctq scorer helps improve translation quality on multiple language pairs and language models. 4. in addition to measures used in the past for example selection, we explore new features. based on our study of various features used, we find that: (a) comet-qe features (learned metrics) are better at example selection than cosine similarity of labse embeddings (task-agnostic semantic matching), (b) similarity of the source input to the example target is more important than its similarity to example source, (c) combining the two observations into a novel feature i.e. using comet-qe based similarity between input source and target is the best feature and is a strong baseline across languages and directions. our code and outputs are accessible via this repository: https://github.com/ai4bharat/ ctqscorer ## related work the selection of an appropriate prompt to enhance machine translation (mt) performance of large language models has been the focus of recent research @xcite @xcite @xcite @xcite . the relevance of the examples to the input sentences is an important factor that affects translation quality. several methods for measuring relevance have been employed: (a) n-gram word overlap between input sentence and examples @xcite , and (b) embedding similarity @xcite @xcite using labse @xcite or roberta @xcite embeddings. the quality of the examples is also an important factor. to ensure quality, examples are either selected from a known high-quality pool @xcite or based on labse or comet-qe @xcite scores between the pairs @xcite . @xcite show that coherency of prompt examples with respect to the test sentence is an important factor for translation performance. @xcite further explore task-level examples, i.e. fixed high-quality examples that result in the best translation quality on a held-out set. zhang et al. (2023 ) study various factors affecting example selection, demonstrating that these features exhibit weak correlation to translation quality, and no single feature can consistently enhance translation quality. however, all these works select examples based on a single feature. in contrast, we propose to combine different features contributing to translation quality for a more informed example selection process. additionally, we investigate some novel features that can influence example selection. ## example selection using multiple features given an input sentence x in source language s, we want to select a set of k examples (e) from an example database (d) to aid in generating the best translation of x into the target language t. in the case of mt, the example database corresponds to a parallel corpus comprising translation pairs (x p , y p ) from which the prompt examples are drawn. our overall approach, as shown in figure 1 is described in this section. an alternative could be to identify n examples (x p ) nearest to x in the embedding space for a better semantic match. however, we opted for the bm25 retriever for efficiency reasons. ## experimental setup we conducted transalation experiments between english and bengali, gujarati, hindi, french, german, and russian. we also studied example selection using different selection algorithms, along with the contextual translation quality (ctq) scorer approach discussed in section 3.1. language iso code dataset #pairs (m) bengali bn samanantar 8.6 gujarati gu samanantar 3.1 hindi hi samanantar 10.1 french fr europarl 1.9 german de europarl 1.8 russian ru paracrawl 5.4 ## conclusion and future work in this work, we propose an example selection method (ctq scorer) that utilizes multiple features for few-shot machine translation with llms. we show that combining multiple sentence-level features to predict the quality of retrieved examples results in significant improvement in translation quality over single features predicting example quality. based on our ablation experiments, we also provide insights into the relevance of various features for example selection. particularly, we would like to highlight that comet-qe based similarity between input source and target is the best feature and is a strong baseline across languages and directions. we also note that the ctqscorer metric can be learnt without the need for a held-out seed parallel corpus. exploration of example selection with multiple features for other nlp tasks is an interesting direction for future research.