Guiding In-Context Learning of LLM s through Quality Estimation for Machine Translation (2024)

Abstract: The quality of output from large language models (LLMs), particularly in machine translation (MT), is closely tied to the quality of in-context examples (ICEs) provided along with the query, i.e., the text to translate. The effectiveness of these ICEs is influenced by various factors, such as the domain of the source text, the order in which the ICEs are presented, the number of these examples, and the prompt templates used. Naturally, selecting the most impactful ICEs depends on understanding how these affect the resulting translation quality, which ultimately relies on translation references or human judgment. This paper presents a novel methodology for in-context learning (ICL) that relies on a search algorithm guided by domain-specific quality estimation (QE). Leveraging the XGLM model, our methodology estimates the resulting translation quality without the need for translation references, selecting effective ICEs for MT to maximize translation quality. Our results demonstrate significant improvements over existing ICL methods and higher translation performance compared to fine-tuning a pre-trained language model (PLM), specifically mBART-50.

## introduction pre-trained large language models (llms) quickly gained popularity (and continue to do so) due to their performance on a large set of natural language processing (nlp) tasks, including machine translation (mt) @xcite . however, the accuracy of their outputs is significantly influenced by the quality of the in-context examples (ices) provided to them @xcite . 1 if these examples do not align well with the specific task and source domain, the llms' outputs can be inaccurate. therefore, there is a critical need to develop (better) methods for selecting appropriate examples that match the task and source domain being translated. these methods collectively fall under the umbrella of in-context learning (icl) @xcite . traditionally, creating ices for mt involves either random selection @xcite or us-ing a strategy such as maximizing an evaluation metric like bleu, to choose examples that improve the metric @xcite . the former was initially used for its simplicity and ease of implementation. however, relying on randomness can lead to inconsistent results and pose significant computational costs @xcite . recent state-ofthe-art (sota) icl approaches focus on retrieving training examples that are closely relevant to the context of source sentences of test sets using unsupervised retrievers, such as bm25 @xcite . recent studies have also shown that a range of factors, such as order @xcite , template @xcite , domain, and number of ices, significantly impact the performance @xcite . naturally, the most effective ices for a given source text are the ones that would impact the resulting translation quality, which would ultimately de-pend on translation references or human judgment. in mt, quality estimation (qe) has become a standard approach for evaluating an mt system's output without relying on reference translations @xcite . recently, @xcite , @xcite , and @xcite showed the effectiveness of domain-specific qe when it comes to domainspecific mt (in contrast to the ineffectiveness of generic qe). building on this and to address the aforementioned challenges, our work proposes to leverage domain-specific qe to assist in the selection of ices, with the goal of determining the suboptimal number and combination of ices to maximize mt quality, all without reference translations. as qe would assess the impact of different ice combinations and sequences, we hypothesize that this integration has the potential to not only improve translation performance but also reduce processing time, as qe could result in smaller sets of ices, which would reduce the inference times @xcite . this is particularly crucial considering the limited number of ices that can be fed into llms @xcite . therefore, our study aims to investigate the feasibility of selecting ices on a per-source basis. specifically, we aim to answer the following research question (rq): how effective are domainspecific qe models in determining ices for translation tasks in an llm? our proposed icl methodology for mt combines an unsupervised retriever to select ices with qe to assess their impact on the translation quality, determining which ice combination to include. instead of feeding all selected examples, we only select examples whose qe points to maximizing the llm translation quality. our findings on german-english translations demonstrate that our proposed approach outperforms the current sota icl methods for mt as well as a fine-tuned mbart-50 @xcite . ## icl using quality estimation for mt to utilize llms for effective mt, as noted in section 1, what is needed is a set of examples to provide the context (and thus guide or steer the llm toward a correct, context-specific translation) --that is, a set of ices --and what is further important is the number of ices and their combination. foot_1 ultimately, what is required is that the ices provide context that is neither too specific nor too broad and can effectively boost the translation. our goal with this work is to develop a methodology that optimizes both these aspects in order to deliver high-quality mt. our methodology for identifying effective ices involves two key components: (1) an unsupervised retriever that locates examples closely related to the sentence to be translated and ( 2 ) a search algorithm that uses qe to select a combination of examples that leads to the improvement of translation quality, i.e., aiming to maximize the bleu score. ## experiments setup we conducted four main experiments to test the effectiveness of our methodology. three of these experiments compare our methodology to existing icl ones in different settings, or modes. the fourth experiment compares our methodology to a fine-tuned mbart-50, aiming to assess which method is preferred (with respect to obtaining better translations). it is important to note that we do not fine-tune the llm. the process of building the qe model used in our experiments is detailed in section 3.2. ## analysis output analysis pre-trained llms often exhibit over-generation, i.e., the generation of a larger number of tokens than expected by a human (in comparison to a reference), necessitating extensive postprocessing (e.g., post-editing) @xcite . figure 1 shows the tokenized output lengths (translations) for our model (mode 1, patience 8), 8 alongside the r-bm25 with 16 examples. the analysis shows that the length distributions for both models align with the reference distribution, suggesting that the models do not over-generate. to quantitatively compare these distributions to the reference, we employed the kolmogorov-smirnov (ks) test @xcite . the results indicate that for r-bm25 versus the reference, the ks statistic is relatively high (0.0749), reflecting a significant difference between the translation lengths of r-bm25 and the reference distribution. the extremely low p-value (2.39 × 10 -5 ) further confirms this significant discrepancy. conversely, for mode 1 with @xmath0, the ks statistic is considerably lower (0.0232), indicating a much smaller difference in translation lengths. the higher p-value (0.6451) suggests no significant difference, implying that the distribution of mode 1, @xmath1. these findings suggest that our proposed methodology could yield translations closer in length to the reference, potentially reducing the need for labor-intensive post-processing efforts and enhancing computational efficiency. ## conclusion we propose a novel in-context learning (icl) methodology for enhancing the translation capabilities of large language models (llms) while optimizing computational resources. our approach leverages domain-specific quality estimation (qe) to guide in-context selection, particularly focusing on determining the suboptimal number and the combinations of in-context examples (ices). this novel strategy moves beyond the conventional reliance solely on translation references from development sets seen in prior methods. we evaluated our approach across different modes and early stopping patience values on the german-to-english it dataset. our experiments consistently showed the superior performance of our methodology, surpassing all prior works across both bleu and comet metrics. our method consistently improves bleu scores, although this comes at the cost of increased computation time. we also investigated the impact of ordering the ices based on their unigram overlap with the source text and found it to be not statistically significant. furthermore, our experiments highlighted the value of icl compared to fine-tuning a pre-trained large model, namely mbart-50. we also highlighted that our method leads to less carbon emissions while achieving better translation performance. in the future, we would like to conduct further research on the impact of our proposed methodology across different language pairs, domains and llms. also, we aim to explore alternative metrics beyond bleu to tailor the selection process, as well as additional features such as bigram, type/token ratio, and length when ordering examples prior to their input into llms. metric generic model specific model the sockets listed in the write array will be watched for whether a write operation blocks the socket.