P r E x M e! Large Scale Prompt Exploration of Open Source LLM s for Machine Translation and Summarization Evaluation (2024)

Abstract: Large language models (LLMs) have revolutionized NLP research. Notably, in-context learning enables their use as evaluation metrics for natural language generation, making them particularly advantageous in low-resource scenarios and time-restricted applications. In this work, we introduce PrExMe , a large-scale Pr ompt Ex ploration for Me trics, where we evaluate more than 720 prompt templates for open-source LLM-based metrics on machine translation (MT) and summarization datasets, totalling over 6.6M evaluations. This extensive comparison (1) benchmarks recent open-source LLMs as metrics and (2) explores the stability and variability of different prompting strategies. We discover that, on the one hand, there are scenarios for which prompts are stable. For instance, some LLMs show idiosyncratic preferences and favor to grade generated texts with textual labels while others prefer to return numeric scores. On the other hand, the stability of prompts and model rankings can be susceptible to seemingly innocuous changes. For example, changing the requested output format from “0 to 100” to "-1 to +1” can strongly affect the rankings in our evaluation. Our study contributes to understanding the impact of different prompting approaches on LLM-based metrics for MT and summarization evaluation, highlighting the most stable prompting patterns and potential limitations.

## introduction the recent success of llms has led to a paradigm shift in nlp @xcite . instructiontuning allows llms to respond to complex task descriptions (prompts) @xcite , including conventional nlp tasks, like automatically evaluating natural language generation (nlg) for machine translation (mt) and summarization. building on this, researchers increasingly use 1 we make our code available: https://github.com/ gringham/prexme llms as evaluation metrics, achieving remarkable performance, sometimes relying solely on in-context learning (e.g. @xcite , i.e., with metrics that only use prompting. such prompting-based metrics require minimal data, making them useful for low-resource evaluation scenarios @xcite and more resource-efficient since they do not require fine-tuning. although many prompting-based metrics have been proposed (e.g. @xcite , structured evaluations across different prompting approaches remain scarce, especially for open-source models. the recent eval4nlp 2023 shared task @xcite addresses this by (1) restricting the usage to selected open-source llms and (2) prohibiting their fine-tuning. while the shared-task submissions offer interesting insights, they focus on only a few distinct prompts, leaving the impact and robustness of prompt variations largely unexplored. in this work, we introduce a systematic prompt exploration for metrics (prexme), expanding on eval4nlp 2023 to provide a much larger, template-based, structured evaluation of the effects different input prompts have on an llm-based metric's correlation with human judgements in mt and summarization evaluation. we formulate the following research questions: rq1 can open-source llms evaluate text generation without fine-tuning and how do they differ from each other (see §4)? rq2 can we identify patterns foot_0 in prompts that lead to a stable performance across different datasets, tasks, and models (see §5)? rq3 how should researchers design prompts for new evaluation scenarios (see §6)? in prexme, we construct hierarchical templates based on approaches such as chain-ofthought (cot) @xcite , zero-shot and retrieval-augmented generation (rag) @xcite . each template gets filled with further subtemplates. for example, we vary the requested output formats, such as numeric scores and textual labels (see §3). this setup amounts to more than 720 templates that we evaluate with 7 llms in a 1st phase. in a 2nd phase, we test the generalizability and performance of the prompts with the best correlations on two further datasets. we make the following key contributions: ✓ we conduct a large-scale analysis of over 6.6m prompts for llm-based metrics in mt and summarization evaluation. this thorough exploration covers various prompting techniques, datasets, tasks, and models, making it, to our knowledge, the most extensive of its kind. ✓ we show that certain prompting patterns are robust and generalizable across different tasks and datasets, with median performance being a good predictor for new settings. for example, some models show a distinctive preference for textual labels, while others yield better results with numeric labels. however, in some cases, even minor prompt changes can significantly impact performance. ✓ our study tackles prompt-based evaluation with open-source llms, targeting scenarios where fine-tuning or access to closed-source llms is not possible. such evaluations are still very scarce but important to make research more accessible, fostering diversity and inclusion. ✓ we systematically test established prompting approaches, including zero-shot, cot and rag, to comprehensively evaluate the performance of recent open-source llms for evaluation metrics. aligning with the recommendations of @xcite , we use multiple prompts per model which mitigates the risk of single prompts disproportionately affecting their performance and ensures a fair comparison. the platypus2-70b model @xcite achieves the strongest performance among the tested llms. we first discuss related work on prompting-based metrics for mt and summarization, then connect our study to research on prompting techniques and stability. prompting-based metrics recent advances in llm-based metrics for nlg often rely on incontext learning to predict the quality of generated texts. surveys by @xcite and @xcite offer comprehensive overviews of these metrics. such prompt-based approaches are often built upon closed-source models or test only a few prompts. for example, gemba @xcite , gemba-mqm @xcite and automqm @xcite evaluate strong prompting approaches for mt evaluation with closed-source models. @xcite use @xcite and two opensource models, to explore one novel prompting approach. in contrast, the eval4nlp 2023 shared task @xcite , considers open-source prompt-based metrics, where participants evaluate mt and summarization using only allowed models without fine-tuning. while eval4nlp yielded interesting techniques, the participants explored a limited range of prompts, leaving a gap in the comprehensive analysis of prompting patterns and consistent llm comparisons. prexme addresses these research gaps by systematically analyzing a much larger set of prompts across comparable experimental settings to ( 1 ) study the robustness of prompts across datasets, open-source models and tasks, and to ( 2 ) identify patterns to guide future prompt-based metrics. prompting techniques over recent years, many successful prompting techniques have been developed (e.g., @xcite . our work primarily builds on established methods like zero-shot cot and rag. @xcite propose emotioninducing prompts to improve llm performance. to our best knowledge, we are the first to analyze this technique for evaluation metrics. inspired by this, we also propose a novel emotion-cot pattern (see §3). @xcite previously evaluated output formats for prompt-based metrics, which we extend with a much broader analysis. other works use hierarchical templates for prompt building (e.g. @xcite and tools like langchain @xcite and dspy @xcite support their implementation. we employ hierarchical templates for structured comparisons among prompting patterns. prompting robustness our grid search across various prompts, datasets and tasks extends research on how llms respond to prompt perturbations. @xcite , @xcite , @xcite and @xcite reveal significant performance variations in tasks like natural language inference and sentiment classification. as a solution, @xcite suggest reporting the full range of results across prompt perturbations, while @xcite and @xcite argue for using multiple templates to increase the reliability of evaluation benchmarks. to our best knowledge, we are the first to explore to which degree these robustness issues affect open-source llm-based metrics and how to select the best prompts. also, by prompting the llms with multiple prompts, we follow @xcite and achieve a stable and fair evaluation of llms as mt and summarization metrics. ## setup in this section, we present the templates and prompting techniques we use to utilize llms as metrics, and we outline the datasets and models that we use for testing. we evaluate llms in a reference-free setting (grading a generated hypothesis based on its source without a reference). 3 the evaluated prompt templates provide a comprehensive evaluation framework for llm-based metrics, covering basic in-context learning, sophisticated reasoning, emotional context, and varying output structures, ensuring a thorough assessment of robustness and adaptability across tasks and datasets. prompt templates we construct prompts as hierarchical templates (see figure 1 ), with large tem-plates built from smaller ones. each prompt is built from: (1) the source text and generated hypothesis text that should be graded, (2) a base prompt, (3) a task description, (4) a format requirement and (5) optionally a one-shot demonstration. the base prompt is the top layer of our prompt hierarchy, incorporating the other components. we test three zero-shot (zs) and corresponding one-shot (os) base prompts: (1) plain zs/os (pzs/pos), (2) zs/os-cot and (3) zs/os-cot-emotion (zs/os-cot-em). pzs plainly presents the newline-separated task description, source, hypothesis and format requirement. zs-cot (ko-jima et al., 2022) asks the model to think step by step before returning its output. lastly, zs-cot-em asks the model to describe its "emotions" before the zs-cot prompt. we include cot due to its success in enhancing prompt-based performance in metrics like @xcite , eaprompt @xcite and gemba @xcite . zs-cot-em examines llm performance variations when prompted to express emotions, inspired by our exploration of emotional prompts (see "task description" below). the os versions of the templates add a demonstration field. to avoid fixating the model on specific reasoning steps, we include a placeholder for os-cot where the model should insert its reasoning. the task description is the instruction to grade the generated hypothesis. @xcite find that instructions evoking certain emotions for humans can enhance llm performance. inspired by this, we experiment with "emotional prompts" in the task description. this approach primarily broadens our grid search through simple paraphrasing but also allows us to study the effect of emotions on llm-based metrics. besides neutral prompts, we include instructions like polite, threatening and sceptical. we create 11 task descriptions ourselves and 13 further descriptions with chatgpt. the format requirement specifies the output format the llm should follow when generating a score, including the score range and whether it should be discrete or continuous. we also include prompts that ask the llm to return textual quality labels. overall, we define 10 format requirements. lastly, we construct the optional os demonstrations with rag. we extract demonstrations from wmt21 @xcite for mt and the factuality dataset rose for summarization @xcite . for each demonstration sample in both category description base prompts pzs: "{task_description} \nsource text: {src} \n{result_type}: {hyp} \n{format_requirement} \nscore: " zs-cot-em: "{task_description} \nsource text: {src} \n{result_type}: {hyp} \n{format_requirement} \nfirst describe your emotions, then think step by step and explain your thought process, finally return your judgment in the format 'judgment: '." os-cot: "{task_description} \n here is an example:\n source text: {ex_src} \n{result_type}: {ex_hyp}\n judgement: <description of reasons>. therefore the score is {ex1_score}\n\n now it is your turn to grade the {result_type}.\n source text: {src} \n{result_type}: {hyp} \n{format_requirement} \n first, think step by step and explain your thought process, then return your judgment in the format 'judgment: '." task desc. neutral: "judge the quality of the following {task_specific_insert}." sceptical: "i'm not sure about this one. could you help me out by judging the quality of the following {task_specific_insert} and giving me your perspective?" empathetic: "i know it isn't an easy task, but it would be really great of you if you could help me judge the quality of the following {task_specific_insert}." format req. 0 or 1: return a discrete score of 0 if the {result_type} has flaws and 1 if it is perfect. catastrophic, indifferent or marvelous: choose whether the {result_type} is either "catastrophic", "indifferent" or "marvelous". mqm-based approaches additionally to hierarchical templates, we test the gemba-mqm prompts @xcite with our selected open-source llms. gemba-mqm, which predicts scores based on the number of present errors weighted by severity, normally uses gpt4. we refer to the open-source implementation as localgemba. ## results in phase 1, we run 6,652,800 zs prompts (720 prompt templates with 7 models on 1320 samples) and 71,280 os prompts (9 "best" prompt templates), with no scores extracted in 12.7% resp. 19.4% of cases; the average score of the prompt template is assigned in these instances. further, in phase 2, we evaluate 5,503,896 zs and 1,308,690 os prompts (9 "best" prompt templates for both), with no scores extracted in 22.3% and 19.4% of cases, respectively. the separate prompting techniques (table group 2.), also using the platypus2-70b model, have weaker correlations than the best prompts of the hierarchical templates. the localgemba mqmbased approach is in the best significance cluster for 3 of 11 tasks and is the best prompting based approach for en-de in wmt23. on the other hand, the baseline prompt dsba is significantly the best on summarization for the eval4nlp test set where it also won the shared task, but not for other tasks. regarding the baselines (table group 3.), xcomet outperforms our llm-based approaches for mt evaluation by a varying margin. for instance, for en-es in the eval4nlp test set, the difference is small and xcomet is in the same siginificance cluster as platypus2-70b. however, for some tasks the performance difference is large, e.g., on en-de in wmt23 xcomet performs 0.14 kendall points better. the strong performance of xcomet for mt evaluation is expected as it (1) is based on the multilingual xlmr-xxl model and (2) fine-tuned for mt evaluation. for summarization, prompting approaches significantly outperform bartscore and xcomet. to revisit rq1, our results show that opensource prompt-based llms, while generally promising, struggle to match the performance of the fine-tuned metric xcomet for mt evaluation. however, llms offer higher versatility across different tasks. unlike xcomet, which is mostly limited to mt evaluation, llms can excel in summarization evaluation with minimal prompt adjustments. additionally, llms seem to demonstrate robustness across tasks even without changing input descriptions; for example, the baseline dsba, designed for summarization, also performs well in some mt evaluation tasks. the prompts in group 1 are built from hierarchical templates, i.e., each presented correlation can have a different format requirement, base prompt and task description. to illustrate the distribution of format requirements, we color correlations of prompts with textual quality labels in orange and those with numeric scores in blue. 7 orca-13b and platypus2-70b were prompted to return numeric scores for all but one reported "best" correlations. on the other hand, llama3-70b, nous-13b and tower-13b were mostly prompted to return textual labels. we also observe consistent patterns in the best prompts per model for the base prompt and, less pronounced, for the task description. for instance, the best prompts for tower-13b always use the zs-cot base prompt, while llama3-70b always uses pzs. details of the prompts of each cell, tie-calibrated accuracy, pearson and spearman correlations, and the scores of the eval4nlp dev set are shown in appendix e. our results indicate that models have idiosyncratic preferences for certain patterns. in §5, we further explore these preferences and their robustness. ## analysis in this section, we answer rq2 and examine the robustness of the template components. best prompting patterns per model and dataset first, we explore the best base prompt, task description and format requirement for each model. we do this by analyzing their prevalence in the 2% of prompts with the highest kendall correlation for each task, a cutoff chosen to represent every task. for instance, figure 2 illustrates the differences in the best base prompts between openorca and tower, two llms with contrasting prompt preferences. while orca favors pzs prompts, tower is better with zs-cot and zs-cot-em. for the format requirement, figure 3 shows that orca prefers scores in the range of -100 to 100, while tower can work better with labels. the pie charts for all further models and the comparison between task descriptions are shown in appendix h. in this comparison between all models, for the base prompts, tower uses zs-cot or zs-cot-em in 86.2%, nous in 44.9%, and platypus2 in 23.9% of its best prompts. all other models use these base prompts in less than 10% of their best prompts. for format requirements, llama3-70b uses textual labels in 90.2% of its best prompts, tower in 80.4%, and mixtral in 80%, whereas orca and platypus2 use them in only 8% and 21.7%, respectively. there is no clear trend for llama3-8b and nous. finally, task descriptions show broader distribution (largely due to their higher number). notably, the "curious" task description is used in over 15% of best prompts for llama3-70b, nous, and llama3-8b. "emphasis" is the most used by platypus2 (17.4%) and "dire warning" is the most used by tower (21.4%). regarding rq2, these results show that models have unaligned preferences for prompting patterns, making it difficult to construct a universally good prompt. however, model-specific patterns can be found 8 and models can be grouped based on their best patterns. for example, one group prefers to return numeric scores and the other textual labels. this behavior may partially stem from shared instruction-tuning data. e.g., orca and platy-pus were partly trained on the same data and prefer to return numeric labels, while both llama3 models prefer textual labels (with llama3-8b to a smaller degree). to analyze whether model-specific preferences hold across datasets, we examine the dataset-wise distribution of the top 2% prompts for each model for all mt tasks, separated by zs vs. os (also see appendix i). if a prompting pattern is stable for all models across datasets, the distribution of the prompts that include the pattern should remain mostly unchanged. indeed, there are prompting patterns whose distribution among top prompts is relatively stable across datasets. as an example, figure 4 shows that the change of distributions between the zs evaluations for the eval4nlp train set for mt and the wmt23 dataset is smaller than 12% for each pattern. as another example, the pzs base prompt ranges between 66.7% and 83% for all datasets. also, the "complex labels" format requirement in phase 2 ranges between 50% to 66.7% for zs and 66.7% to 83.3% for os. this does not hold for the phase 1 evaluation, where the template selection was much broader. also, for some prompt patterns, e.g. the "emphasis" and "collaborative" task descriptions, the occurrence in the top prompts seems to swap between datasets. this experiment shows that prompts are to some degree stable between datasets. in the next paragraph, we further examine this stability between global explanations @xcite of the models. datasets, prompting patterns and models. prompt stability next, we quantify how stable the performance of a prompting pattern a is when the dataset, model or other prompt components change. to do so, we compute the rankings of prompts that use a before and after the change and then test the similarity of rankings. for example, we rank format requirements on dataset 1, then we change the dataset and obtain a second ranking. if the first and second ranking are similar, the performance of different format requirements is stable between the two datasets. we test this similarity with the kendall correlation. the ranking of a prompting pattern can be computed in several ways, since multiple templates contain each pattern. in our example, each format requirement has multiple evaluated prompts per dataset, varying by base prompts, task descriptions and tasks. the performance of a specific format requirement in the ranking can, for example, be determined by aggregating its different scores across base prompts, task descriptions, etc. with the mean or median. we test the following aggregation methods: mean, median, mean of top 10%, max, min and saturation @xcite . thereby, we determine that the aggregation with the median leads to the most stable ranking, i.e. the highest kendall correlation between rankings. specifically, we test this by comparing every selection of two aggregation measures in a permutation test (e.g. median vs. mean, mean vs. max, etc.); see appendix §g. for our example, for each format requirement on dataset 1, we compute the median score of all combinations of base prompts, task description and task. then, we do the same for the second dataset and check the correlation of the resulting rankings. a high correlation of the rankings then indicates that the median performance for all prompts using the format requirement is a good indicator of its relative performance on a new dataset. figure 5 shows heatmaps for the stability of the format requirement and task description when the base prompt is changed (see appendix j for further heatmaps). the highest stability is given when changing from pzs to zs-cot or vice versa (0.65). that means, there is a high chance that the format prompt with the highest median correlation will perform good for zs and zs-cot. for the task description a change from zs to zs-cot is unlikely to retain the ranking, underlining the previous paragraph's finding that the format requirement is more stable than the task description. we can also apply this method to quantify the stability of model rankings when switching from pattern a to pattern b. figure 6 shows this analysis for the format requirement. for example, if all models are prompted with "0 to 100" and with "-100 to 100" the ranking of models will not change much. with a change from "simple labels" to "complex labels" the model ranking will change strongly. regarding rq2, the heatmaps highlight that even small changes to the input prompt can drastically impact the relative ranking of llms and other prompting patterns. this aligns with recent studies highlighting the susceptibility of llms to single prompts (e.g. @xcite @xcite . however, the heatmaps also show that not every change to the input has this effect and they can be used as in-dicators for the transferability of new prompting patterns. ## recommendations we now address rq3 and give recommendations to employ open-source prompt-based metrics. among the evaluated models, platypus2-70b demonstrates superior performance. for 13b models, tower and orca exhibit the highest correlations in mt and summarization tasks. we recommend to use the prompting patterns that most frequently yield top correlations for these models (refer to §5 and appendix h). when introducing a new prompting pattern or model, its median performance across other prompting patterns can serve as an indicator of the pattern's efficacy in new contexts. thereby, the actual predictive power of the median (or other aggregation measures) for each dimension can be determined based on previous evaluations. the results and source code of prexme provide a foundational basis for this analysis. ## conclusion we introduce prexme, a large-scale study of prompting templates for open-source nlg metrics. evaluating 720 templates and over 6.6m prompts, we offer recommendations for enhancing metric robustness. further, prexme acts as a benchmark of recent open-source llms as metrics for mt and summarization. 9 limitations one limitation of our work is that even though we evaluate a large variety of possible prompts, there is still a lot of interesting possible variety in prompting approaches that we did not explore for now (e.g., the detail level of task instructions or structured output formats). a further limitation is that we cannot be sure that the newer llm models did not see parts of the older datasets in their training data. also, the selection of the best prompts that are presented in the result tables is currently based on the maximum instead of the median, which was found to highlight the most stable prompts. generally, by selecting the 9 "best" prompts for phase 2 we are narrowing the search space. hence, the interplay between prompt patterns might not be fully represented for these phases. furthermore, our heatmaps only compare one dimension, while another is changed, possibly simplifying the interplay between the others. as another limitation, in rare cases the context size of the models was exceeded. future work could explore different ways to handle this than cutoff. further, the heatmaps show many kendall correlations and may be prone to statistical effects for some values. lastly, we assume that localgemba is performing worse than, e.g., pzs prompts because of its higher prompt complexity, while the original gembamqm can handle it due to gpt4 being more advanced. however, we did not test pzs prompts with gpt4 to confirm it performs worse than gembamqm there.