When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024)

Abstract: Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value — we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a *hybrid* scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at [https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness](https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness).

## introduction the advent of transformer-based large language models (llms) @xcite @xcite @xcite has led to a generational leap in generative models, enabling interaction with computing devices through natural language. this advancement encompasses improvements that have rendered many earlier benchmarks and leaderboards obsolete @xcite , leading to the compilation of more challenging and comprehensive tests. however, the current generation of leaderboards still does not satisfy many of * core contributor † corresponding author: haidark@sdaia.gov.sa the requirements of researchers and practitioners looking to build on llms @xcite . since llms are extremely expensive to both train and inference, selecting the llm (or llm training recipe) is often the most costly decision for the entire project. stable leaderboards are critical to making the right decision. leaderboards based on multiple choice questions (mcq) for evaluation @xcite @xcite @xcite present both convenience and significant limitations @xcite . while mcqs offer an automated and quantifiable means to assess certain aspects of model ability (e.g., knowledge), they fall short as a stable means to measure performance. figure 1 demonstrates the instability of the leaderboard ranking of one popular benchmark, massive multitask language understanding (mmlu) @xcite , under small perturbations. moreover, the reliance on mcqs raises concerns about the models being overfit to these benchmarks, potentially excelling in structured tests while lacking real-world applicability. this discrepancy highlights the need for more holistic and diverse evaluation methods that transcend the simplicity of mcqs @xcite . it also prompts critical reflection on how these models might inadvertently be trained to achieve high scores through spurious correlations, pattern recognition, and optimization for specific question formats rather than genuine language comprehension or knowledge. as llms continue to evolve, it is imperative to develop evaluation frameworks that can more accurately assess their abilities in a way that mirrors the complexity of real-world use. despite being widely used, benchmarking with mcqs has turned out to be anything but simple. it requires the full synchronization of eval-figure 1 : minor perturbations cause major ranking shifts on mmlu @xcite . models can move up or down up to eight positions on the leaderboard under small changes to the evaluation format. columns (from left): 1) original ranking given by mmlu using answer choice symbol scoring (a common default). 2) ranking under an altered prompt for the same questions, where answer choice symbols are replaced with a set of rare symbols. 3) setting where the correct answer choice is fixed to a certain position (in this case, b). 4) using the cloze method for scoring answer choices. under each new ranking, we report @xcite with respect to the original ranking (lower k τ indicates more disagreement between rankings) uation frameworks and results often vary wildly due to nuanced differences. for example, minor changes in prompting and scoring can produce invalid results for particular llms foot_0 . recent studies have investigated the issue of sensitivity in evaluating llms, with some demonstrating that llms are susceptible to the ordering of answer choices and bias towards specific tokens/symbols @xcite @xcite , and others exploring different prompt modifications and their effects in benchmarking llms @xcite @xcite . in this work, we conduct a broad range of minor perturbation experiments to mcq benchmarks and observe the disruption it causes to model rankings on leaderboards. we also take additional steps to precisely identify the limitations of llms on this measurement approach. the contributions of this paper can be summarized as follows: ## llm evaluation with mcqs evaluating llms with mcqs has rapidly become a standard for measuring the knowledge and reasoning capabilities of the model @xcite @xcite . many such mcq benchmarks have been used to measure llms, including massive multitask language understanding (mmlu) (hendrycks et al., 2020), ai2 reasoning challenge (arc) @xcite , and common-sense question answers (csqa) @xcite . mechanically, testing llms with mcqs is accomplished by presenting the question along with the answer choices to the model and selecting the choice deemed most probable by the model. although this setup appears straightforward, llms react in unpredictable ways to formatting and other minor changes to the questions or the answers. llm performance on an mcq test can change with the introduction of an extra space (e.g., between the question and answer) or adding an additional instructional phrase (e.g., "choices:"). in addition to this brittleness, @xcite found changes to the order in which answer choices are presented to gpt4 and instructgpt can change the model's prediction. these findings lead us to take a deeper look at how mcq-based benchmark results are affected by small perturbations to question formats, llm prompts, presentation of few-shot examples, and other dimensions. in particular, we introduce variations in three categories: our main aim is to quantify how these small perturbations/variations change the rankings of a set of models on a particular benchmark. as mcq benchmarks-based leaderboards are often used to compare models and guide model selection, we investigate the robustness of benchmarks for this purpose. figure 1 demonstrates how existing benchmarks exhibit significant undesirable shifts in rankings under small perturbations. ## methods in this section, we describe and justify the perturbations we apply in each category. we note that some mcq test changes, like modifying the order of answer choices, can change performance even for humans but the effect is typically not pronounced @xcite . in general, our modifications are designed to be small perturbations to the mcq and prompts that should not affect performance. the exception to this is some of the in-context knowledge manipulations described in section 3.3, which are designed to improve or degrade performance drastically. ## experiments in the bulk of our experiments, we focus on the mmlu benchmark due to the extensive nature of our experiments (11 models, 22+ settings), and extend some experiments to arc-challenge to show generalizability. mmlu @xcite ) is a commonly used benchmark for comparing llms, consisting of 57 subjects spanning four domains: humanities, stem, social sciences, and others. each subject includes at least 100 multiple-choice questions with 4 answer choices. the entire benchmark contains 14,042 questions @xcite and a.30 have a breakdown of the mmlu subjects and their distributions). ai2 reasoning challenge (clark et al., 2018b) is a benchmark consisting of 7787 grade school science questions. the benchmark is split into two sets: easy and challenge. we conduct experiments on the challenge set (arc-c) which is proven to contain harder questions for existing models. the questions in arc-c have 3-5 answer choices. unless otherwise stated, the reported score for each experiment/model combination on mmlu is the mean accuracy across all 14,042 questions. all tested model tokenizers encode the multiple-choice answers as single tokens. hence, the accuracy is equivalent to the normalized accuracy. all baseline and modified mmlu benchmarks were performed using the lm evaluation harness @xcite library. their implementation of mmlu measures the log-likelihood of each of the answer tokens ['a', 'b', 'c', 'd'] after the input prompt and chooses the letter with the highest probability as the model's answer. some of our experiments require permuting the answer choice order ( for each variation introduced to the mcq benchmarks, we calculate the change in accuracy (∆acc) and recall standard deviation (rstd) for each model. rstd measures the bias of a model to a particular answer choice by computing the standard deviation of recalls for each answer choice @xcite . this metric quantifies how much the model favors particular positions for the correct answer choice. we typically observe whether rstd changes (∆rstd) are significant across experimental settings. to measure the change in ranking induced by an applied perturbation to a benchmark, we measure the normalized kendall's τ distance between two rankings of n models @xcite . kendall's τ computes the number of swapped pairs between two rankings normalized by the total number of pairs n(n-1) ## conclusion building robust leaderboards is a major challenge for the community, as leaderboards help practitioners select the best methods and models for continued research. given this importance, it is critical to address the breakdown of existing leaderboards to the slight perturbations we demonstrated in our work. in addition to building our understanding of the causes of this sensitivity (e.g., bias in llms and bias in scoring methods), future work should aim to adopt and design benchmark practices that avoid these pitfalls. board sensitivity. to explain llm bias, we attempted to design experiments that isolate each source of bias under mcq but were unable to quantify the relative effects of bias or conclude why they occur. this was further complicated by our inability to access the pretraining datasets of the llms to rule out benchmark contamination. future work in this direction will most likely require tools from interpretability research (e.g. mechanistic interpretability). one of our main contributions was to highlight where mcq-based leaderboards fail to deliver stable rankings. although we succeeded in showing this, we were unable to demonstrate a robust solution to this problem. our recommendation to, for example, use hybrid scoring methods is still not completely robust to perturbations.