Precise Model Benchmarking with Only a Few Observations (2024)

Abstract: How can we precisely estimate a large language model’s (LLM) accuracy on questions belonging to a specific topic within a larger question-answering dataset? The standard direct estimator, which averages the model’s accuracy on the questions in each subgroup, may exhibit high variance for subgroups (topics) with small sample sizes. Synthetic regression modeling, which leverages the model’s accuracy on questions about other topics, may yield biased estimates that are too unreliable for large subgroups. We prescribe a simple yet effective solution: an empirical Bayes (EB) estimator that balances direct and regression estimates for each subgroup separately, improving the precision of subgroup-level estimates of model performance. Our experiments on multiple datasets show that this approach consistently provides more precise estimates of the LLM performance compared to the direct and regression approaches, achieving substantial reductions in the mean squared error. Confidence intervals for EB estimates also have near-nominal coverage and are narrower compared to those for the direct estimator. Additional experiments on tabular and vision data validate the benefits of this EB approach.

## introduction accurate evaluation of large language models (llms) is crucial for identifying their strengths and weaknesses. while broad topics like math or history often have lots of data available for testing, specific topics may lack suitable test data. for example, we may not be able to collect queries for niche subjects, such as a particular historical event, legal terms from specific jurisdictions, rare medical conditions, or regional dialects. consequently, we may fail to reliably assess the llm's understanding of these topics. when gathering more data is figure 1 : estimates of llm accuracy and their 95% confidence intervals for predictions made by gemma-2b across various subgroups on a subset of hellaswag. the empirical bayes estimates have precision similar to the direct estimator for some subgroups (e.g., clean and jerk) and higher precision for others (e.g., high jump). this approach also provides tighter confidence intervals for the estimates; e.g., see running and sports topics. to illustrate this problem, consider a randomly sampled subset of hellaswag @xcite , a dataset with multiple-choice questions on various topics or subgroups; e.g., see figure 1 . our goal is to precisely estimate the accuracy of the answers given by the llm in each subgroup using only the limited data available. traditionally, we estimate model performance via a direct estimator (dt), which computes the average accuracy in each subgroup separately. however, when a subgroup has only a few questions, the dt estimates can exhibit high variance and become unreliable. one may expect the performance of the llm on related subgroups to be associated. synthetic estimation aims to exploit this relationship via regression modeling (sr) @xcite . for example, we can incorporate the llm's knowledge of sports to aid in estimating its accuracy on questions about running. the resulting estimates will generally exhibit less variance compared to direct estimation but may be biased, and thus they may be imprecise for subgroups with large sizes. empirical bayes (eb) approaches combine dt and sr estimates, adjusting the contribution of each for every subgroup separately @xcite . in theory, they hold the promise to improve the precision of these baselines @xcite and allow for the construction of confidence intervals @xcite ; e.g., see again figure 1 . however, to the best of our knowledge, an extensive empirical evaluation of the validity of this approach for precise model benchmarking is missing. in this work, we empirically assess the precision of eb estimates of llm performance on subgroups (domains, tasks, topics, etc.) across multiple benchmark datasets. our results in section 4 show that eb estimates are consistently more precise than those of dt and sr. their confidence intervals have good coverage and substantially smaller widths than those of dt. additional experiments on vision and tabular data validate these findings (appendix c). ## problem setup we say that estimators with lower average mse are more precise. by standard arguments, the mse for each subgroup g decomposes into bias and variance components: mse( in the following, we consider three estimation methods. ## results in this section, we present the core set of results, conveying our key takeaways. we defer to appendix c for additional experiments, including a comparison with other baselines and other data modalities. ## discussion our investigation contributes to a growing body of work on model evaluation in presence of resource constraints @xcite @xcite . our results show that an eb estimator, which combines dt and sr estimates, consistently provides more precise estimates of subgroup-level model performances compared to dt and sr alone. when subgroups are large, the estimates of eb align with those of dt. the uncertainty of eb estimates can also be effectively quantified via confidence intervals, which achieve near-nominal coverage and are consistently tighter than those derived for dt estimates (although see limitations in appendix b). overall, eb proves to be a simple and useful method for model evaluation when estimating performance for lots of subgroups with (many or) few observations. this serves as an appendix to the paper "precise model benchmarking with only a few observations." the content of the appendix is organized as follows.