Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks (2024)

Abstract: Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs). The research community often relies on a model’s average performance across the test prompts of a benchmark to evaluate the model’s performance. This is consistent with the assumption that the test prompts within a benchmark represent a random sample from some real-world distribution of interest. We note that this is generally not the case; instead, we hold that the distribution of interest varies according to the specific use case. Hence, we analyze the robustness of LLM benchmarks to their underlying distributional assumptions. We find that (1) the correlation in model performance across test prompts is non-random, (2) accounting for correlations across test prompts can change model rankings on major benchmarks, (3) explanatory factors for these correlations include semantic similarity and common LLM failure points.

## introduction since the introduction of the transformer architecture @xcite , large language models (llms) have progressed into sophisticated systems with an outstanding ability to comprehend and generate text that mimic human language. notable models in this domain include chatgpt 1 , utilizing the gpt-3.5-turbo or gpt-4 architectures 2 , llama @xcite , chatglm @xcite , alpaca @xcite , and falcon @xcite . due to their effectiveness, llms are becoming very popular in both academia and industry, making their evaluation crucial. however, this effectiveness comes at the cost of increased complexity, which makes their evaluation very challenging. although prior research has introduced benchmarks for different tasks along with evaluation measures, † these authors contributed equally to this work. 1 new chat: https://chat.openai.com/ 2 models -openai api: https://platform.openai. com/docs/models/ these assessments often overlook potential biases. when a benchmark includes multiple prompts with similar characteristics, it can increase or decrease the average performance of a model, so model comparisons can become brittle with respect to benchmark composition (see figure 1 for an illustrative example). in this work, we show that the inherent connections between the prompts in current benchmarks impact the models' performance and their relative rankings. the standard approach for evaluation on a benchmark is to (i) obtain model responses for each prompt in the benchmark, (ii) compute the performance metrics for each response, (iii) aggregate (usually average) the performance metrics to obtain a single performance metric over the benchmark, and (iv) compare models by comparing their aggregate performance. when aggregating performance metrics in step iii above, each prompt is generally weighted equally @xcite . however, using equal weights reflects the assumption that prompts in the benchmark are "equal", in the sense that prompts are representative samples of a target distribution of interest. in the case of llms, the notion of a target distribution (i.e., the distribution of all possible prompts for a given use case) is usually not well-defined. for example, different natural language inference (nli) applications may have very different target distributions, and we should not expect a single benchmark to capture every one. therefore, one must ask: what distribution do the prompts in the benchmark represent? would considering different distributions fundamentally change model comparisons? in this work, we present a novel approach to assess the robustness and adequacy of benchmarks used in evaluating llms, by analyzing the performance of multiple llms on a set of four major benchmarks. our key contributions are outlined below: in figure 1a , each benchmark prompt contributes equally to the model evaluation. in contrast, figure 1b accounts for correlations between prompts and the weights of the prompts are adjusted accordingly during evaluation. in scenario 1a, the red llm ranks highest because it excels in math, and the benchmark is biased towards math tasks (7 out of 12 prompts are math-related). when considering different weights in scenario 1b, we observe a different ranking outcome. ## related work evaluating the performance of llms has become a critical area of research, drawing significant attention in recent years. comprehensive surveys of llm evaluation and benchmark quality can be found in @xcite ; @xcite ; @xcite . when assessing the quality of llms, the robustness aspect is becoming of increasing importance @xcite . robustness investigates the stability of a model when confronted with unforeseen prompts. robustness research can be divided into four main lines of work @xcite : (i) robustness under distribution shift @xcite , (ii) robustness to adversarial input @xcite , (iii) robustness to prompt formats, including instruction templates @xcite @xcite , and (iv) robustness to dataset bias @xcite @xcite . our work falls into the latter category. reducing bias on benchmarks is a long-standing area of research spanning many diverse fields. applications range from weighing survey responses to match a target population @xcite , to accounting for language biases in visual questionanswering @xcite . in the context of nli, researchers have looked into improving the quality of prompts in order to mitigate certain types of biases. work in this area has focused on determining the quality of prompts by generating optimal prompts @xcite or by clustering prompts based on semantic similarity @xcite . additionally, researchers have investigated data leakage between benchmarks and llm training data @xcite . limited research has been conducted to study inherent biases in llm benchmarks. among existing works, @xcite and @xcite have shown that models leverage spurious statistical relationships in the benchmark datasets and, thus, their performance on the benchmarks is overestimated. in the same spirit, le @xcite propose to investigate aflite @xcite , an iterative approach to filter datasets by removing biased data points to mitigate overestimation of language models' performance. more recently, @xcite show that performance of llms is highly sensitive to minor changes in benchmarks with multiple-choice questions. other studies demonstrate that benchmarks often include redundancy and effective llm evaluation can be achieved with a significantly smaller sample size @xcite . our work is orthogonal yet complementary to previous work. in particular, we propose a new method to identify biases in a benchmark by looking at the performance of multiple recent llms on that benchmark. we show that similarity in performance correlates with similarity in prompts. to the best of our knowledge, our work is the first approaching benchmark biases by analyzing and leveraging the performance of a collection of models on a set of major benchmarks; as well as investigating the impact of inherent distributional biases in benchmarks used on llm comparative studies. ## proposed method in this section, we outline the problem setup and introduce the notation and expressions that will be employed throughout the paper. second, we present the approach to evaluate whether relationships between prompts (based on models' performance) are statistically non-random. furthermore, we describe our method for analyzing how sensitive model comparisons are with respect to different distributional assumptions of the benchmark. finally, we present our proposed methodology for exploring the origins of relationships between prompt performance vectors. ## experimental setup in this section, we describe the setting of our experiments. specifically, we provide details on the benchmarks and evaluation metrics we use, the llms we consider, and how we evaluate performance of the llms on the benchmarks. ## results in this section, we present the results of the experiments described in section 3 on the benchmarks. ## conclusion and future work llms are commonly evaluated on benchmarks that may include multiple prompts testing similar skills. in this work, we demonstrate this bias on major benchmarks, by showing that model performance across different prompts is significantly correlated. furthermore, we demonstrate that llm comparative studies can be significantly altered when using non-uniform weights for prompts during evaluation. the suggested approach can serve as a consistency check in comparative studies of llms, ensuring that the results take into consideration benchmark biases. finally, we show that similar model performance across prompts can be explained by semantic similarity, but is most likely derived from common failure points of the llm. our findings could influence a larger diagnostics tool for evaluating the robustness of model quality comparisons with respect to distributional assumptions of benchmarks. future work also in- p e rs im m o n 8 b l la m a -2 1 3 b k o a la 1 3 b m is tr a l 7 b v ic u n a 1 3 b c la u d e -2 c h a tg p t-tu rb o -b a se -0 5 1 6 fa lc o n -1 8 0 b l la m a -2 7 0 b c h a tg p t-tu rb o -0 3 0 1 g p t-4 -t u rb o -1 1 0 6 (c h a t) g p t-4 -0 3 1 4 (c h a t)