Team NLLG submission for E val4 NLP 2023 Shared Task: Retrieval-Augmented In-Context Learning for NLG Evaluation (2023)

Abstract: In this paper, we propose a retrieval-augmented in-context learning for natural language generation (NLG) evaluation. This method allows practitioners to utilize large language models (LLMs) for various NLG evaluation tasks without any fine-tuning. We apply our approach to Eval4NLP 2023 Shared Task in translation evaluation and summarization evaluation subtasks. The findings suggest that retrieval-augmented in-context learning is a promising approach for creating LLM-based evaluation metrics for NLG. Further research directions include exploring the performance of various publicly available LLM models and identifying which LLM properties help boost the quality of the metric.

## introduction like any machine learning task, the nlg problem requires a quality metric to compare model outputs to a gold standard. the most popular method for human evaluation is mqm @xcite , which allows building an interpretation of the generation model through error detection. however, this technique requires expensive manual work of an expert. as a consequence, automatic evaluation systems that would have a high correlation with state-of-the-art evaluation techniques, in particular mqm, would be highly desirable as a replacement for human mqm annotations. one such approach, became entrenched after the appearance of llms, is zero-shot or few-shot generation by text query, prompt. the score is obtained from the model by (i) the numerical estimate itself @xcite , (ii) aggregation over the probabilistic distribution of the model @xcite or (iii) a real function over the resulting text, repeating the existing methodology of expert evaluation @xcite . the shared task of eval4nlp 2023 @xcite challenges to solve the problem of evaluating machine translation and summarization results using a fixed set of llms without any finetuning techniques and in a reference-free manner. reference-free means that the metric rates the provided machine translation solely based on the provided source sentence/paragraph, without any additional, human written references. the shared task has the following goals: the main judgement metric during the competition is segment-level kendall-τ correlation between model scores and mqm expert annotations. for the second goal listed above, the organizers will evaluate explanations manually. the following list of models from huggingface @xcite was available during the competition: ## related work in general, designing high-quality evaluation metrics for nlg tasks such as summarization and machine translation is an highly active field of research. it is especially active since the recognition that decades old metrics such as bleu @xcite and rouge @xcite are inadequate for evaluation @xcite @xcite . the focus in recent years is on developing high-quality llm based metrics @xcite that are (among others) explainable @xcite @xcite @xcite , efficient @xcite @xcite , robust @xcite , and reproducible @xcite . the focus of eval4nlp's shared task is on explainable high-quality metrics induced from prompting the most recent classes of llms including variants of llama @xcite . the ability of @xcite to solve different nlg problems in a zero-shot manner led to appearance of new nlg evaluation approaches utilized this model. gemba @xcite used a set of instruction prompts for machine translation evaluation which differ from each other with score ranges and its descriptions, model is expected to generate repeatedly the text until it is a score as sequence of digits. another usage of gpt-4 in nlg evaluation is g-eval (liu et al., 2023), which used a similar approach for summarization evaluation with zero-shot instruction based generation but with another score obtaining. the final score is an aggregation of digits with their token generation probabilities. automqm @xcite is a finegrained approach which allows to construct interpreted evaluation via modeling mqm metric. the model is expected to generate error major and minor spans, after that the deterministic score based on mqm error weights is alculated. the vanilla approach used full transformer architecture, we try to repeat this approach with decoder-only model. similarly to our proposed approach, retrievalaugmented in-context learning was used for multiclass text classification in @xcite . in this paper, the pretrained retrieval model from sen-tencetransformers @xcite is used to collect the in-context examples, closest to the input text. in their case, the length of the examples is consistently small, so they are able to fit as many as 110 in-context examples by greedily selecting examples until they completely fill the model's context window. ## approach the basis of our approach is the selection of several few-shot examples for each specific instance. to do this, we use an index, a large array of source texts from the training dataset for different language pairs. in the index, all texts are stored as embeddings, which are then compared to the source text by the cosine distance. we specifically compare samples by their source texts, as we hypothesize that for similar source examples, the way to evaluate translation/summarization is usually similar. the whole workflow with the examples is illustrated on figure 2 . the few-shot examples themselves use the same prompt format as the request -only with an already inserted score. all the examples go in a row, forming a single prompt from several few-shot parts and a prompt with the requested rating. for a more accurate assessment, we obtain various examples from the index, both with high and low scores. ## results & discussion the results of evaluation of the proposed approaches are presented in from the multilingual perspective, all our models rely on substantially limited/non-existent multilingual pretraining of the base model as well as the fine-tuned versions. in fact, all those models use the small vocabulary of 32k tokens. this does seem to be enough to capture word pieces for english and similar latin scripted languages: spanish and german. however, in the case of the english-chinese language pair, we see a consistent drop in metric correlation among all tested llms. lastly, the fine-grained approach described above yielded only 0.11 on kendall-τ correlation with human judgment for the english-german translation subtask. while we were not able to finish its inference on other mt subtasks in time, we did find several problems with this approach. in most cases, the model failed to accurately produce spans for identified errors as they contained some words from the translated text but in a disarranged order, along with unrelated words. also, we found that in some cases, the model generated a list of duplicate or near duplicate errors, which resulted in an overly pessimistic approximation of the translation quality. we hypothesize that it was likely due to the model we have used. in the original paper @xcite , the authors use google's private palm-2 @xcite model which is a) has more (540b) parameters, b) was pre-trained on 'parallel data covering hundreds of languages' and c) is based on encoder-decoder architecture. in contrast, in our case, the largest model had only 70b parameters and was mostly pretrained on monolingual english data. also, according to @xcite , the decoder-only causallms are suboptimal for the case of in-context learning, while prefixlms (encoder-decoder) are better suited to utilize in-context examples for generating prediction. ## conclusion during experiments for eval4nlp 2023 shared task, we considered approaches with in-context learning and fine-grained evaluation and observed that adding reference examples could boost the generation result, even though it is the only score. however, this method is sensitive to the encoder model with index setup, examples' set size and requires a lot of diverse references. we did not manage to observe good results for fine-grained approach with automqm, we think that the problem is with the model size and architecture. some ideas for further research include: a) exploring the capabilities of llms with more parameters when applied with our prompting strategy, b) utilizing models with larger (or unlimited) context window to increase the number of in-context examples, c) experimenting with llms pre-trained on multilingual data for translation evaluation and d) applying encoder-decoder llms to achieve better incorporation of in-context examples.