Efficient Benchmarking (of Language Models) (2024)

Abstract: The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are associated with massive computational costs, extending to thousands of GPU hours per model. However, the efficiency aspect of these evaluation efforts had raised little discussion in the literature.In this work, we present the problem of Efficient Benchmarking, namely, intelligently reducing the computation costs of LM evaluation without compromising reliability. Using the HELM benchmark as a test case, we investigate how different benchmark design choices affect the computation-reliability trade-off. We propose to evaluate the reliability of such decisions, by using a new measure – Decision Impact on Reliability, DIoR for short.We find, for example, that a benchmark leader may change by merely removing a low-ranked model from the benchmark, and observe that a correct benchmark ranking can be obtained by considering only a fraction of the evaluation examples.Based on our findings, we outline a set of concrete recommendations for efficient benchmark design and utilization practices. To take a step further, we use our findings to propose an evaluation algorithm, that, when applied to the HELM benchmark, leads to dramatic cost savings with minimal loss of benchmark reliability, often reducing computation by x100 or more.

## introduction given the ongoing advances in the versatility and performance of language models (lms), they are now expected to perform a diverse range of tasks. this expectation raises a profound challenge -how do we evaluate and rank the quality of different lms over a variety of capabilities? figure 1 : helm model ranks for different numbers of inference calls. each colored curve represents a different model. model ranks are extremely stable even when compute drops dramatically: a ×10 decrease in the number of examples per scenario produces nearly the same results as the full benchmark, while a ×400 reduction still clusters models in the same small groups seen in the full compute regime. in this work, we call attention to the topic of efficient benchmarking, namely intelligently reducing the computation costs of evaluation without compromising reliability. while the trade-off between computation and performance is usually discussed in the context of pre-training (e.g., scaling laws; @xcite and finetuning (e.g., parameter efficient; lialin et al., 2023), here we call for putting this trade-off on the center stage of evaluation design. in practice, the compute side of the trade-off already plays a role in most large-scale evaluation decisions, both in benchmark design @xcite and in its use for evaluation (e.g., choosing the number of seeds; @xcite . however, despite their practical importance, these choices and their impact on benchmark reliability have hardly been discussed in the literature, making researchers apply their own efficiency heuristics instead of using systematic guidelines or literature when building their benchmarks. in order to advance efficient evaluation practices, the community is in need of a systematic set of guidelines and recommendations. these, in turn, must be based on a rigorous study of the different decisions made in benchmark design and how they affect efficiency. to begin addressing these challenges, we propose decision impact on reliability -dior -a way to measure the impact of a decision over a setup size (e.g., 1k examples, 10 datasets) on the reliability. in addition, we perform a comprehesive analysis study on efficient benchmarking. with helm @xcite as a test case, we test various decisions made and how they affect the trade-off between computation and reliability: decisions about scenarios which are aggregated phenomena ( §5.1), subscenarios ( §5.2), few-shot prompts ( §5.4) and the metric ( §5.5). among other findings, we observe a substantial computation redundancy (see ## experimental setting in our main experiments, we calculate dior to examine the reliability under the current realization of the benchmark, as well as more efficient realizations. thus, we calculate dior for varying amounts of compute, ranging from the full helm benchmark to a small fraction of it (e.g., a benchmark with 1 scenario, or 100 examples). for each design choice (number of examples, scenarios etc.) we sample different instantiations of this choice, and use them to calculate dior. we follow a bootstrap approach, namely, sampling 1k times with repetition. for example (in §5.1), to test whether taking 10 scenarios reliably indicates the best model, we sample 10 scenarios (out of the available 16) 1k times, calculating the win rate values for each sample (in a sample, some datasets may be chosen more than once, or not at all).