State of What Art? A Call for Multi-Prompt LLM Evaluation (2024)

Abstract: Recent advances in LLMs have led to an abundance of evaluation benchmarks, which typically rely on a single instruction template per task. We create a large-scale collection of instruction paraphrases and comprehensively analyze the brittleness introduced by single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. We find that different instruction templates lead to very different performance, both absolute and relative. Instead, we propose a set of diverse metrics on multiple instruction paraphrases, specifically tailored for different use cases (e.g., LLM vs. downstream development), ensuring a more reliable and meaningful assessment of LLM capabilities. We show that our metrics provide new insights into the strengths and limitations of current LLMs.

## introduction recent years have seen an explosion of large language models (llms), which generalize to unseen tasks via natural language instructions. various llm evaluation benchmarks, such as big-bench and helm, use a single instruction template per task, evaluating all models against it @xcite . however, there could be a myriad of ways to phrase an instruction template for a given task; see figure 1 for examples of different templates for the task of recognizing homophones. naturally, llm performance depends on the chosen template. we explore the question of robustly comparing different models on a given task. we first create a dataset of paraphrased instructions, employing three automatic paraphrasing methods based on recent techniques such as chain-of-thought. we manually verify and filter a large collection of more than 175 paraphrases for different tasks (5k instruction paraphrases in total), which we make publicly available for future research. 1 next, we use our dataset to perform a large-scale statistical evaluation of over 6.5m instances, involving 20 different llms and 39 tasks from 3 benchmarks. we find that models perform very differently on different instruction paraphrases. for example, figure 1 shows four models evaluated on four semantically equivalent prompts, with both absolute and relative performance varying widely; one can even observe cases where the same model performs the best on one instruction and the worst on a semantically equivalent instruction (e.g., gpt-3.5-turbo on p 1 vs. p 4 ). subsequently, we argue that very little can be said on either absolute or relative performance based on single-instruction evaluation. this may also partially explain why some models seem less accurate in practice than their formal evaluation suggests. note that while the claim that evaluating against a single instruction template leads to brittle results is not surprising per se, to the best of our knowledge it has never been subjected to rigorous empirical testing before. to address the limitations of single-instruction evaluation, we propose to take a step back and consider multi-prompt llm evaluation -a set of metrics which measure aggregated performance over a set of instruction template paraphrases. we argue that different use cases should entail different evaluation metrics. for example, llm developers may be interested in measuring despite all instructions being semantically equivalent, both absolute performance and relative ranking vary widely. the robustness of performance across multiple instruction templates. in contrast, developers aiming to integrate an llm into a specific downstream task may be interested in comparing models according to their corresponding top-performing instruction. we evaluate 20 llms with our metrics, finding that their absolute and relative performance differ from results obtained with the benchmarks' original instructions. we demonstrate that different models excel in different metrics: for instance, in the lmentry benchmark, llama-based models are comparable to t5-based models when looking at top-performing instructions, but lag behind when average performance is considered, due to poor performance on a large number of paraphrases. we also show that our automatic paraphrasing method is effective, and there is no need to manually verify the paraphrases. our results suggest that future work should use multi-prompt llm evaluations and choose a metric for aggregating the results according to the extrinsic needs of the evaluators. we hope that our work will help spur more consistency and comparability in llm evaluation, which is strongly tied to real-world usage of llms. ## background and definitions below we survey how generalization to a new task format is evaluated and compared between llms, finding that the common practice involves a single (or very few) task instruction templates. in the rest of the paper, we will argue that such practice leads to brittle, unreliable results. 2024 ), we separate between task instruction, samples, and inputoutput exemplars which may be provided during in-context learning. we define an instruction template for a given task as a string with placeholders where the input samples are to be inserted. as seen in figure 1 , the same task can be described using different task instruction templates. ## evaluating against a single prompt leads to instability in results as discussed in the previous section, llms are usually evaluated against a single instruction template. in this section, we will show that this approach is quite brittle. indeed, a simple rephrasing of the instruction template can lead to drastic changes in both absolute and relative model performance. in section 4.1 we create a large number of automatically generated instruction paraphrases for tasks from the lmentry and bbh benchmarks. paraphrases are created using an llm and verified by human annotators. in section 4.2, we statistically analyze the performance of various llms against these instruction templates and quantify the variation in model performance. finally, in section 4.3, we show that models exhibit similar brittleness with manually written paraphrases for tasks from the bbl benchmark. ## different use cases merit different metrics we have shown that llm performance is greatly affected by paraphrasing of instruction templates. this calls into question current evaluation practices, which typically rely on llm performance on a single instruction template. in this section we explore ways to evaluate llms using a diverse set of instruction templates. most importantly, we argue that the answer should depend on the purpose of the evaluation, and that different extrinsic needs should lead to different evaluation metrics, rather than striving for a coarse catch-all metric. we introduce a set of metrics, each tailored to specific scenarios and realistic user needs. notations. in the following, m is a pretrained llm, @xmath0, y i )} denotes an evaluation dataset for m , i t is a set of natural language task instruction paraphrases for t (e.g., obtained via automatic paraphrasing), and ε(m, t, i) ∈ [0, 1] denotes the aggregated performance of m on samples from t , using a single instruction template @xmath1, e.g., accuracy or f 1 . ## multi-prompt evaluation in figure 6 we evaluate all our 16 models according to the metrics we proposed in the previous section, on sample tasks from each of the three benchmarks (full results for all tasks are available in our repository). we report several interesting observations. first, we find that all aggregate metrics diverge from the performance on the original instruction templates. for the vast majority of the tasks in our study, the top three models determined by the original instruction templates were different from those which ranked first according to the average and maximum metrics. more broadly, model ranking depended on the metric used. for instance, see figure 6 (top): in lmentry's rhyming word task, falcon-instruct-7b and vicuna-13b rank first according to maxp (0.74, gray and yellow bars), but their average performances avgp are only 0.17 and 0.15, respectively. similarly, across all tasks in the lmentry benchmark, llama-based models were competitive with t5-based models in terms of maxp . however, in terms of avgp , they tended to lag behind, due to extremely poor performance on a large number of paraphrases (see figure 5 for %paraphrases that achieved at least 5% accuracy). finally, we found that noise stemming from automatic paraphrase generation has virtually no impact on metric-based model rankings. we compute kendall's τ to compare model rankings before and after the manual filtering of paraphrases. the results ( ## small-scale evaluation of openai models on prompt paraphrasing in this section we perform a small-scale evaluation showing that api llms are also sensitive to benchmark max perf. average perf. combined perf. lmentry .963 .978 .948 bbh .991 .983 .966 ## conclusions our research highlights the sensitivity of large language models (llms) to prompt paraphrasing, challenging the adequacy of single-prompt evaluations. we propose alternative evaluation metrics that use a diverse set of instruction templates for each task, designed for more robust and meaningful llm evaluation. for example, llm developers may be interested in measuring the robustness of performance across multiple prompts, which we propose to evaluate as the average across a large collection of prompts. in contrast, when developing a downstream model, different models should be compared according to their corresponding top-performing prompt. evaluating based on these metrics underscores the necessity for nuanced evaluation methods, revealing notable differences in absolute performance and relative model rankings compared to traditional evaluations. we hope that our work will help spur more consistency and comparability in llm evaluation which is strongly coupled to real-world llm uses. we believe this shift is crucial for accurately understanding and leveraging the true capabilities of llms.