The E val4 NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics (2023)

Abstract: Generative large language models (LLMs) have seen many breakthroughs over the last year. With an increasing number of parameters and pre-training data, they have shown remarkable capabilities to solve tasks with minimal or no task-related examples. Notably, LLMs have been successfully employed as evaluation metrics in text generation tasks. Strategies employed in this context differ in the choice of input prompts, the selection of samples for demonstration, and the methodology used to construct scores grading the generations. Approaches often differ in the input prompts, the samples that are selected for demonstration and the construction process of scores from the output. Within this context, we introduce the Eval4NLP 2023 shared task that asks participants to explore such approaches for machine translation evaluation and summarization eval- uation. Specifically, we select a list of allowed LLMs and disallow fine-tuning to ensure a focus on prompting. We test the approaches of the participants on a new reference-free test-set spanning 3 language pairs for machine transla- tion as well as a summarization dataset. Further, we present an overview of the approaches taken by the participants, present their results on the test set and analyze paths for future work. Fi- nally, as a separate track, we perform a human evaluation of the plausibility of explanations given by the LLMs and its effect on model performance. We make parts of our code and datasets available.

## introduction the chatgpt revolution in late 2022 has ignited a wide public and scientific debate about the possibilities (and limitations) of generative ai in various fields and application scenarios (leiter et al., 1 https://github.com/eval4nlp/sharedtask2023/tree/main 2023b; @xcite , including education (halaweh, 2023), logic @xcite , medicine @xcite , math @xcite , programming @xcite and science @xcite . the immense research interest has also triggered the exploration of numerous approaches that leverage generative large language models (llms) as evaluation metrics @xcite @xcite @xcite for natural language generation (nlg) tasks like machine translation (mt) and summarization. recent llm based approaches differ, for example, in their prompting strategies, e.g., in the way that natural language instructions are used to trigger the llm to compute metric scores. for example, gemba @xcite uses zero-shot prompting to directly predict scores or quality labels in the output. in contrast, automqm @xcite instructs llms to predict fine-grained error labels and uses these to compute the final scores. these works have contributed to the exploration of prompting for nlg evaluation, but an exhaustive exploration of approaches remains unaddressed. further, many approaches leverage closed source llms while much fewer use open source llms. those approaches relying on open source llms put a large focus on acquiring training data (e.g. @xcite and fine-tune models to specific tasks. given this typical focus on fine-tuning and motivated by promising work on prompting techniques foot_0 (e.g. @xcite figure 1 : using a generative llm as mt evaluation metric. in this example, the metric is reference-free. i.e. it grades the translated sentence based on its source sentence. the input sentences are wrapped into a prompt that is given to an llm. the llm generates an output and a final score could for example be constructed from this textual output or from other values involved in the process. the red borders indicate the focus of our shared task. participants should evaluate the best prompts and the best approaches to construct scores from model output. 2022; @xcite @xcite , we notice a research gap in the thorough examination of prompting and score composition in the domain of nlg metrics, especially for open-source generative llms. the eval4nlp 2023 shared tasks aims to fill this gap by disallowing participants to fine-tune models and by restricting model usage to a fixed list of llms (see figure 1 ). hence, participants may only vary how models are prompted, how scores are extracted, and how models are used in combination. to make the task more inclusive, we consider large and small(er) llm's in two separate tracks. this is different from shared tasks without model restriction, where the largest models often perform best, for example, the wmt metrics shared task (e.g. @xcite . the goal of the shared task is to design evaluation metrics for mt and summarization, which we select as sub-tasks of nlg, while adhering to the model restrictions. our contributions are the following: our paper is structured into 8 sections. §2 gives an overview of how our shared task is related to other competitions. §3 describes the competition setup and §4 / §5 describe the datasets and annotation process for the test phase respectively. in §6, we highlight the approaches tested by the participants, especially those for the test set submissions. §7 presents the final scores of the participants on the test set and further analyses. finally, §8 discusses future work and provides a conclusion. ## related work in this paragraph, we describe other work that is related to our shared task. in specific, we give a brief overview of evaluation metrics, highlight the recent development on metrics that are based on generative llms and describe related shared tasks. ## datasets during the dev-phase of our shared task, we provided participants with a train-and a dev-set. for the test-phase, we further added a test-set. train-& dev-set our train-and dev-sets are constructed from two datasets. for mt, we select the en-de and zh-en mqm partitions of the wmt 2022 metrics shared task @xcite . for summarization, we select summeval @xcite . we conduct our task in a reference-free setting, that is, we do not provide human written reference translations or summaries. hence, we remove the references provided with wmt and summeval. summeval has separate scores for relevance, factuality, coherence and consictency for each sample. we construct a single score per example by averaging these separate scores. further changes to the original datasets include the split into train-and dev-partitions as well as shuffling. in the dev phase participants could experiment with generalizable (prompting) approaches. test-set we collect a novel test set for the testphase of our shared task. it consists of 3 language pairs for mt: en-de, en-es, en-zh and a summarization part. we only choose high-resource languages, as the llama(2)-based models have seen limited multilingual data during their pre-training and fine-tuning. on parts of the test set, we gather wikipedia articles created after 15.07.23 as source texts. 13 figure 3 shows the score distributions of our datasets. we can see that all language pairs exhibit a pattern of centering around values divisible by 5. this makes sense, as mqm weighs major errors with 5 points. also, in en-es, samples have generally received a higher score; i.e., fewer major errors were annotated. finally, our summarization dataset, which uses a combined annotation scheme (see §5) does not show this pattern. ## annotation in this section, we describe the annotation process of our dataset. for mt annotation, we hire one annotator per language pair: one master student who speaks spanish as mother tongue with english certifications, one nlp bachelor student, who is a native english speaker that lives in germany since many years, and one data and discourse studies master student, who is a native chinese speaker who uses english on a daily basis. for summarization annotation, we hire one nlp bachelor student as well as a data and discourse studies master student with a prior master in linguistics. both annotators annotated the same data. all annotators demonstrated their suitability for the role in initial test rounds with further applicants. the distribution of our final mt dataset is shown in we use google's anthea 14 as annotation tool, because of its support for mqm annotations @xcite ). as we mostly annotate single sentences for mt, we modify anthea to provide context via a wikipedia url that can be consulted if annotators are unsure about a translation. for summarization, annotations were 13 limitations of this approach are discussed in §8 14 https://github.com/google-research/ google-research/tree/master/anthea conducted in a modified version of anthea with a new template (we show a screenshot of the ui in appendix c). for both data sets, we perform fine-grained annotations. in mt this has been shown to yield more reliable human annotations than other annotation schemes @xcite . also, the fine-grained annotations could be used later-on to verify automatically generated explanations. as we only received 2 submissions for the explainability track, we do not consider apply this in this report. mt we construct the mt dataset from random source sentences with a minimum length of 110 characters, as tokenized by the nltk sentence tokenizer 15 . in a few cases, multiple sentences are concatenated due to missing spaces between dots. we obtain machine translations with 4 different translation models (see to verify the quality of the dataset, members of our team who are native speakers of the respective 17 https://en.wikipedia.org/wiki/wikipedia: content_assessment 18 https://languagetool.org/de target languages have annotated small subsets of 30-50 samples of the datasets. ## shared task approaches the test phase of our shared task received submissions from 12 different teams, 9 of which submitted system papers. here, we summarize the approaches of these 9 systems and announce their final standings. we divide the approaches taken by the participants into probability-based, output-based and agent-based. 28 besides their final approaches, the participants have explored a large number of possible variations. afterwards, we introduce the baseline approaches, we compare the participants with. probability-based probability-based approaches calculate how likely a paraphrase or translation of an input is generated with an llm. probability based approaches are explored by @xcite and @xcite . @xcite define 10 different prompts to translate a source sentence with an llm. they combine this approach with demonstrating samples in the input prompt selected by (among others) sbert @xcite . further, they use ensembles to recombine the scores of multiple prompts and models. @xcite use the probability-based approach with own prompts and prompts designed by the authors of gptscore @xcite . output-based all submitted papers explore the direct usage of an llm's natural language output as score. @xcite test the same sample selection and ensembling strategies described above with 4 different prompts in an output-based setting. @xcite follow a similar approach to @xcite and retrieve demonstration examples by finding similar examples with labse @xcite embeddings in an output-based setting. @xcite try one approach in which they present a prompt that triggers the prediction of a single score and one approach that triggers the model to first rate summary qualities for consistency, coherence, fluency and relevancy. then they aggregate these scores in 3 different ways. @xcite quantize orcamini themselves to run an even smaller model (which is close to violating the allowed settings of the shared task). they provide a detailed explanation to their model that triggers it to produce fine-grained scores and a combined score in the same output. @xcite choose rating guidelines from related work -concretely, the human guidelines (hg) for summeval, the machine guidelines for g-eval @xcite and evaluation steps generated by gpt4 (openai, 2023). they test various adaptations to this prompt, explore the usage of examples in the prompt and the usage of coarse-grained vs. fine-grained and aggregated scores. on the test set, they add a shortcut for very bad summarizations and employ bucketing for their scores. @xcite explore evaluating 6 different criteria over all model combinations. @xcite explore 8 prompt types: 3 base prompts and their extensions with chain-of-though @xcite , zero-shot and few-shot settings. mahmoudi (2023) explores various zero-shot and few-shot settings with orcamini. finally, @xcite ; @xcite generate explanations as an additional request to their model. agent-based while they also use an outputbased setup, we place lu and yu-ting (2023) in a separate group. they define 4 characters that should be played by a model and a list of 10 properties. for example they define "internet troll" as a critical character or "teacher" as more knowledgeable character, with the intention that different viewpoints can help to judge generation quality better. then, they evaluate the combined 40 settings and use xgboost @xcite to combine their scores. while they did not add their top submissions to the final leaderboard they present their reasonably good final scores in their paper. baselines as baselines, we use the widely used metrics bertscore (with xlmr-large embeddings) @xcite , sbert @xcite cosine-similarity (with xlmrlarge embeddings), supert @xcite , gemba (kocmi and federmann, 2023) and team authors tracks pradhan/todi (pradhan and todi, 2023) s, su kotonya et. al. (kotonya et al., 2023) s, su dsba (kim et al., 2023) s, l, su hit-mi&t lab (zhang et al., 2023) s, mt iust_nlp_lab (mahmoudi, 2023) s, su, e ltrc (baswani et al., 2023) s, mt, su, e nllg (larionov et al., 2023) l, mt, su taiwansenior (lu and yu-ting, 2023) s, mt iml (akkasi et al., 2023) s, l, su comet-kiwi-xxl @xcite . further, we include one baseline for every allowed model that uses the da score prompt of gemba (kocmi and federmann, 2023) (with a slight modification for summarization). the models are further specified in appendix d. ## results and analysis in this section, we first report statistics of the shared task. then we will present and discuss the final system ranking. note that we include submissions of participants on the test-set-leaderboard that did not submit a system paper. however, we do not describe their approaches in §5. lastly, we will discuss the implications of these results on the development of generation-based metrics. statistics the dev-phase on codalab has received 44 registrations, 13 of which have submitted their scores. in total, there have been 1048 submissions on the dev-set suggesting that some participants might have optimized their method on the dev-set. especially, one participant submitted 417 submissions on the dev set. the test-phase on codabench has received 21 registrations and 248 submissions from 11 participants. we have restricted the number of allowed submissions per day to 10. allowing a higher number would enable participants to optimize their approaches on the test-set too much, such that the results would not reflect the generalization capability anymore. on the other hand, we wanted to give participants the option to try out multiple approaches they designed. further, codabench would sometimes fail to compute scores and still deduct one submission. hence, 10 submissions per day allows us to continue in these cases. two participants have used up a contingent of ≈ 50 submissions. of the 11 test-phase participants, 9 have submitted a system paper. the first authors are from china, india (2), korea, taiwan, canada, iran, germany and the united kingdoms. that means, many authors are from developing countries. also, many authors are students. hence, their resource availability was limited, leading many of them to opting for smaller models. correlation with humans here, we present the results that the participants achieve on the test sets. a mapping between team names and authors can be found in performance the best performing approaches of the participants achieve a similar kendall correlation as our team members when we were testing the inter-annotator agreement on a small subset of samples (see §3). this suggests that these approaches are already close to the performance of native speakers with little training with the annotation process (as compared to our main annotators with a strong language background and more annotation experience on the task). this is an intriguing finding and highlights the potential of current open source models with and without fine-tuning. especially, as many prompting approaches, like treeof-thoughts or self-refinement still remain to be explored. further, it shows that for closed source models like chatgpt or gpt4 similar opportunities may exist and lead to new state-of-the-art metrics. the results also show that comparably small hardware can already be enough to create strong new metrics. explainability only 2 participants @xcite have submitted entries with complementary explanations to the codabench leaderboard. both directly prompted the model to give reasoning for the model's decision. thus, we perform the human experiment on explainability only on a small scale of 50 annotations for randomly selected samples of our summarization dataset. two annotators of our team were presented with source, summary, mqm annotations (to help to identify problems), the scores of the participants and the explanations of the participants. they annotated which of two explanations they pre-fer. one annotator preferred explanations of one system, lets call it a, in 27 cases and explanations of the other in 23 cases. the other annotator preferred system a in 24 cases and the other system in 26 cases. in these annotations the annotators agree in 56% of cases. these findings show that the annotators did not have a clear preference between the systems. also, we notice that many explanations tend to be vague and return texts such as "the summary has a good coherence and fluency". in some cases, the explanations correctly describe problems. we show one example explanation of @xcite in ## conclusion we discuss future work and then summarize the shared task in a conclusion.